Image processing device and method, data processing device and method, program, and recording medium

ABSTRACT

In an image processing device and method, program, and recording medium of the present invention, high frequency components of a low quality image and a high quality image included in a studying image set are extracted, and an eigenprojection matrix and a projection core tensor of the high frequency components are generated in a studying step. In a restoration step, a first sub-core tensor and a second sub-core tensor are generated based on the eigenprojection matrix and the projection core tensor of the high frequency components, and a tensor projection process is applied to the high frequency components of an input image to generate a high quality image of the high frequency components. The high quality image of the high frequency components is added to an enlarged image obtained by enlarging the input image to the same size as an output image.

TECHNICAL FIELD

The present invention relates to an image processing device and method,a data processing device and method, a program, and a recording medium,and particularly, to an image processing technique suitable forrestoring, interpolating, enlarging, and encoding high image qualityinformation that does not exist in image data (low image qualityinformation) before processing.

BACKGROUND ART

A technique is proposed as a method of generating a high resolutionoutput image from a low resolution input image, in which pairs of lowresolution images and high resolution images of a multiplicity of imagecontents are studied in advance, a conversion (projection) relationshipfrom low resolution information to high resolution information isobtained, and the projection relationship is used to generate (restore)an image including the high resolution information from the lowresolution input image

(Non-Patent Literature 1)

The conventional method can be divided into a studying step and arestoration step. In the former studying step, the projectionrelationship between the low resolution information and the highresolution information of the pair group (will be called a “studyingimage set”) of the low resolution images and the high resolution imagesare studied in advance using a tensor singular value decomposition(TSVD). For example, a tensor is obtained, the tensor indicating theprojection relationship between modality eigenspaces, such as conversionfrom a real space of low resolution pixels to a pixel eigenspace andconversion to a personal difference eigenspace (eigenspace) of person,as well as conversion to a high resolution pixel eigenspace andconversion from the high resolution pixel eigenspace to the real space.

Meanwhile, in the restoration step, the studied tensor is used toproject input images of arbitrary low resolution information includingthe studying image set onto images of high resolution information.

According to the technique, the number of variations of modalities (suchas individual difference between people, expression of face, resolutionof image, face direction, illumination change, and race) of projectionconversion can be expressed by the order of the tensor (studying modelcan be designed accordingly), and the projection that satisfies inputconditions allows highly accurate restoration.

CITATION LIST Non-Patent Literature

-   {NPL 1}-   JIA Kui, GONG Shaogang “Generalized Face Super-Resolution”, IEEE    Transactions of Image Processing, Vol. 17, No. 6, June 2008 Page.    873-886 (2008).

SUMMARY OF INVENTION Technical Problem

However, in the conventional technique, since input conditions of theprojection conversion are harsh, and particularly, the tolerance for theillumination change is narrow, there is a problem that the restoredimage quality after the projection is deteriorated when an image thatdoes not meet the conditions is input. An example of a method of solvingthe problem includes adding the illumination change as a modality of theprojection conversion. However, if the modality is added, the projectionfunctions defining the projection relationship increase, and theprocessing time of the projection conversion increases.

The problem relates not only to image processing, but also to variousdata processing using similar projection conversion, such as speechrecognition, language data processing, living body informationprocessing, and natural/physical information processing.

For example, in an application to the speech recognition, a samplingfrequency, the number of quantization bits (the number of bits), and thelike of voice data can be the modalities, and a studying eigenspace forspeech recognition needs to be prepared for each sampling frequency,such as 48 kHz, 44.1 kHz, and 32 kHz, and each number of quantizationbits, such as 16 bits and 8 bits.

In an application to the language processing, a studying eigenspace forlanguage recognition needs to be prepared for each language, such asJapanese and English. In an application to the living body informationprocessing, the natural/physical information processing, and the like, astudying eigenspace for information processing needs to be prepared foreach of sampling frequency or needs to be prepared for each number ofquantization bits.

The present invention has been made in view of the circumstances, and anobject of the present invention is to provide a highly robust (strong)image processing device and method and a program that can alleviateinput conditions of an image as a conversion source and that can obtainan excellent conversion image for an image with an illumination change.Another object of the present invention is to provide an imageprocessing technique that can reduce the capacity of the memory to beused and reduce the processing load to speed up the process. Anotherobject of the present invention is to provide a data processing deviceand method, a program, and a recording medium that expand and apply theimage processing technique to a general data processing technique.

Solution to Problem

The following aspects of the invention are provided to attain theobjects.

A first aspect of the present invention provides an image processingdevice characterized by including: information acquisition means foracquiring an eigenprojection matrix generated by a projectioncomputation from a studying image group including at least one of animage pair formed by high frequency components of a first-quality imageand a second quality image with different image qualities and an imagepair formed by the high frequency components and medium frequencycomponents of the first-quality image and the second-quality image andacquiring a projection core tensor generated from the studying imagegroup and the eigenprojection matrix; first sub-core tensor generationmeans for generating a first sub-core tensor corresponding to acondition specified by a first setting from the acquired projection coretensor; second sub-core tensor generation means for generating a secondsub-core tensor corresponding to a condition specified by a secondsetting from the acquired projection core tensor; filtering means forgenerating a low frequency component control image in which highfrequency components or the high frequency components and mediumfrequency components of an input image to be processed are extracted;first subtensor projection means for projecting the low frequencycomponent control image by a first projection computation using theeigenprojection matrix and the first sub-core tensor to calculate acoefficient vector in the intermediate eigenspace; second subtensorprojection means for projecting the calculated coefficient vector by asecond projection computation using the second sub-core tensor and theeigenprojection matrix to generate a projection image from the lowfrequency component control image; image conversion means for generatinga conversion image with an image quality different from the input image;and addition means for adding the projection image and the conversionimage.

According to the first aspect, the image processing device that obtainsa high quality output image from a low quality input image suppressesthe low frequency components of the input image to execute a high imagequality formation process by the tensor projection. As a result, theinfluence of image degradation in the high image quality formationprocess by the tensor projection due to a disturbance or noise, such asan illumination change, included in the low frequency components can beremoved from the output image, and robustness (strength) for the lowfrequency components (such as a disturbance and noise) can be increasedin the restored high quality image.

Limiting the target of the projection conversion to the high frequencycomponents, or the medium frequency components and the high frequencycomponents, instead of all frequency components, can allocate the entireeigenspace, which can be used to generate the studying image group, tothe high frequency components or the medium frequency components and thehigh frequency components.

It is preferable to include storage means for storing the acquiredeigenprojection matrix and the projection core tensor. The storage meansmay be non-volatile storage means, such as a hard disk, an optical disk,and a memory card, may be storage means for temporary storage, such as aRAM, or may be a combination of the storage means.

The first setting can designate a projection relationship for projectingthe first-quality image on the intermediate eigenspace, and the secondsetting can designate a projection relationship for projecting thesecond-quality image on the intermediate eigenspace.

A second aspect of the present invention provides an image processingdevice characterized by including: information acquisition means foracquiring an eigenprojection matrix generated by a projectioncomputation from a studying image group including at least one of animage pair formed by high frequency components of a first-quality imageand a second quality image with different image qualities and an imagepair formed by the high frequency components and medium frequencycomponents of the first-quality image and the second-quality image,acquiring a first sub-core tensor corresponding to a condition specifiedby a first setting, the first sub-core tensor generated using aprojection core tensor generated from the studying image group and theprojection matrix, and acquiring a second sub-core tensor correspondingto a condition specified by a second setting, the second sub-core tensorgenerated using the projection core tensor; filtering means forgenerating a low frequency component control image in which highfrequency components or the high frequency components and mediumfrequency components of an input image to be processed are extracted;first subtensor projection means for projecting the low frequencycomponent control image by a first projection computation using theeigenprojection matrix and the first sub-core tensor to calculate acoefficient vector in the intermediate eigenspace; second subtensorprojection means for projecting the calculated coefficient vector by asecond projection computation using the second sub-core tensor and theeigenprojection matrix to generate a projection image from the lowfrequency component control image; image conversion means for generatinga conversion image with an image quality different from the input image;and addition means for adding the projection image and the conversionimage.

A third aspect of the present invention provides the image processingdevice according to the first or second aspect, characterized in thatthe information acquisition means acquires an eigenprojection matrixgenerated by the projection computation from the studying image groupincluding the image pair formed by the high frequency components of thefirst-quality image and the second-quality image and acquires aprojection core tensor generated from the studying image group and theeigenprojection matrix, the filtering means generates a high frequencycomponent image in which the high frequency components of the inputimage are extracted, and the first subtensor projection means, which isfor projecting the low frequency component control image by the firstprojection computation using the eigenprojection matrix and the firstsub-core tensor to calculate the coefficient vector in the intermediateeigenspace, and the second subtensor projection means generate aprojection image of the high frequency components from the highfrequency component image to generate image information of a highfrequency area exceeding a frequency area expressed in the input image.

According to the aspect, a high frequency area that is not expressed inthe input image can be expressed in the output image.

A fourth aspect of the present invention provides an image processingdevice characterized by including: eigenprojection matrix generationmeans for generating an eigenprojection matrix generated by a projectioncomputation from a studying image group including at least one of animage pair formed by high frequency components of a first-quality imageand a second quality image with different image qualities and an imagepair formed by the high frequency components and medium frequencycomponents of the first-quality image and the second-quality image;projection core tensor generation means for generating a projection coretensor defining a correspondence between the high frequency componentsand an intermediate eigenspace or between the high frequency componentsas well as the medium frequency components and the intermediateeigenspace of the first-quality image and a correspondence between thehigh frequency components and the intermediate eigenspace or between thehigh frequency components as well as the medium frequency components andthe intermediate eigenspace of the second-quality image; first sub-coretensor acquisition means for generating a first sub-core tensorcorresponding to a condition specified by a first setting from thegenerated projection core tensor; second sub-core tensor acquisitionmeans for generating a second sub-core tensor corresponding to acondition specified by a second setting from the generated projectioncore tensor; filtering means for generating a low frequency componentcontrol image in which high frequency components or the high frequencycomponents and medium frequency components of an input image to beprocessed are extracted; first subtensor projection means for projectingthe low frequency component control image by a first projectioncomputation using the eigenprojection matrix and the first sub-coretensor to calculate a coefficient vector in the intermediate eigenspace;second subtensor projection means for projecting the calculatedcoefficient vector by a second projection computation using the secondsub-core tensor and the eigenprojection matrix to generate a projectionimage from the low frequency component control image; image conversionmeans for generating a conversion image with an image quality differentfrom the input image; and addition means for adding the projection imageand the conversion image.

A fifth aspect of the present invention provides the image processingdevice according to the fourth aspect, characterized in that theeigenprojection matrix generation means generates the eigenprojectionmatrix by the projection computation from the studying image groupincluding the image pair formed by the high frequency components of thefirst-quality image and the second-quality image, the projection coretensor generation means generates the projection core tensor from thestudying image group and the eigenprojection matrix, the filtering meansgenerates a high frequency component image in which the high frequencycomponents of the input image are extracted, and the first subtensorprojection means, which is for projecting the low frequency componentcontrol image by the first projection computation using theeigenprojection matrix and the first sub-core tensor to calculate thecoefficient vector in the intermediate eigenspace, and the secondsubtensor projection means generate a projection image of the highfrequency components from the high frequency component image to generateimage information of a high frequency area exceeding a frequency areaexpressed in the input image.

A sixth aspect of the present invention provides the image processingdevice according to any of the first to fifth aspects, characterized inthat the high frequency components and the medium frequency componentsof the first-quality image are extracted by applying the same process asthe filtering means to the first-quality image, and the high frequencycomponents and the medium frequency components of the second-qualityimage are extracted by applying the same process as the filtering meansto the second-quality image.

According to the aspect, the high frequency components or the mediumfrequency components of the studying image group for generating theeigenprojection matrix and the first and second projection tensors andthe high frequency components or the medium frequency components of theinput image applied with a process using the eigenprojection matrix andthe first and second projection tensors are extracted by the sameprocess. Therefore, a projection image and a conversion image suitablefor the addition by the addition means are generated.

A seventh aspect of the present invention provides the image processingdevice according to any of the first to sixth, characterized by furtherincluding weighting factor determination means for determining weightingfactors for weighting the projection image and the conversion imageadded by the addition means.

It is preferable in the aspect to determine the weighting factorsaccording to the reliability of the restoration of the tensor projectionprocess.

An eighth aspect of the present invention provides the image processingdevice according to any of the first to seventh, characterized in thatthe filtering means executes a process of extracting components greaterthan a frequency based on a Nyquist frequency in the input image.

In the aspect, the filtering means functions as a high frequencycomponent pass filter (high-pass filter).

A ninth aspect of the present invention provides the image processingdevice according to any of the first to eighth aspects, characterized inthat the first-quality image is an image with a relatively low imagequality of the image pair, the second-quality image is an image with arelatively high image quality of the image pair, and the change qualityimage is an image with a higher image quality than the input image.

A tenth aspect of the present invention provides the image processingdevice according to any of the first to ninth aspects, characterized inthat the first setting designates a projection relationship ofprojecting the first-quality image on the intermediate eigenspace, andthe second setting designates a projection relationship of projectingthe second-quality on the intermediate eigenspace.

An eleventh aspect of the present invention provides the imageprocessing device according to any of the first to tenth aspects,characterized in that the projection computation is one of localitypreserving projection (LPP), locally linear embedding (LLE), and lineartangent-space alignment (LTSA).

A twelfth aspect of the present invention provides the image processingdevice according to any of the first to eleventh aspects, characterizedin that the studying image group includes the image pair targeting theface of a person, and the intermediate eigenspace is a personaldifference eigenspace.

A thirteenth aspect of the present invention provides the imageprocessing device according to any of the first to twelfth aspects,characterized by further including: first feature area specifying meansfor specifying first feature areas from an inputted image; compressionprocessing means for compressing image parts of the first feature areasin the inputted image at a first compressive strength and compressingimage parts of other than the first feature areas at a secondcompressive strength which is a compressive strength higher than thefirst compressive strength; and image quality change processing meansfor projecting at least the first feature areas by the first subtensorprojection means and the second subtensor projection means to change theimage quality.

A fourteenth aspect of the present invention provides the imageprocessing device according to any of the first to thirteenth aspects,characterized in that the projection computation includes a projectioncomputation using a local relationship.

According to the aspect, the medium frequency components or the highfrequency components that are easily lost in global information in thePCA or the like are easily preserved as a result of the conversion bythe projection preserving a local structure as in the LPP or the like.Therefore, a new advantageous effect that there is a possibility offurther improving the restored image quality is obtained.

A fifteenth aspect of the present invention provides an image processingmethod characterized by including: an information acquisition step ofacquiring an eigenprojection matrix generated by a projectioncomputation from a studying image group including at least one of animage pair formed by high frequency components of a first-quality imageand a second quality image with different image qualities and an imagepair formed by the high frequency components and medium frequencycomponents of the first-quality image and the second-quality image andacquiring a projection core tensor generated from the studying imagegroup and the eigenprojection matrix; a first sub-core tensor generationstep of generating a first sub-core tensor corresponding to a conditionspecified by a first setting from the acquired projection core tensor; asecond sub-core tensor generation step of generating a second sub-coretensor corresponding to a condition specified by a second setting fromthe acquired projection core tensor; a filtering process step ofgenerating a low frequency component control image in which highfrequency components or the high frequency components and mediumfrequency components of an input image to be processed are extracted; afirst subtensor projection step of projecting the low frequencycomponent control image by a first projection computation using theeigenprojection matrix and the first sub-core tensor to calculate acoefficient vector in the intermediate eigenspace; a second subtensorprojection step of projecting the calculated coefficient vector by asecond projection computation using the second sub-core tensor and theeigenprojection matrix to generate a projection image from the lowfrequency component control image; an image conversion step ofgenerating a conversion image with an image quality different from theinput image; and an addition step of adding the projection image and theconversion image.

A sixteenth aspect of the present invention provides an image processingmethod characterized by including: an information acquisition step ofacquiring an eigenprojection matrix generated by a projectioncomputation from a studying image group including at least one of animage pair formed by high frequency components of a first-quality imageand a second quality image with different image qualities and an imagepair formed by the high frequency components and medium frequencycomponents of the first-quality image and the second-quality image,acquiring a first sub-core tensor corresponding to a condition specifiedby a first setting, the first sub-core tensor generated using aprojection core tensor generated from the studying image group and theprojection matrix, and acquiring a second sub-core tensor correspondingto a condition specified by a second setting, the second sub-core tensorgenerated using the projection core tensor; a filtering process step ofgenerating a low frequency component control image in which highfrequency components or the high frequency components and mediumfrequency components of an input image to be processed are extracted; afirst subtensor projection step of projecting the low frequencycomponent control image by a first projection computation using theeigenprojection matrix and the first sub-core tensor to calculate acoefficient vector in the intermediate eigenspace; a second subtensorprojection step of projecting the calculated coefficient vector by asecond projection computation using the second sub-core tensor and theeigenprojection matrix to generate a projection image from the lowfrequency component control image; an image conversion step ofgenerating a conversion image with an image quality different from theinput image; and an addition step of adding the projection image and theconversion image.

A seventeenth aspect of the present invention provides an imageprocessing method characterized by including: an eigenprojection matrixgeneration step of generating an eigenprojection matrix generated by aprojection computation from a studying image group including at leastone of an image pair formed by high frequency components of afirst-quality image and a second quality image with different imagequalities and an image pair formed by the high frequency components andmedium frequency components of the first-quality image and thesecond-quality image; a projection core tensor generation step ofgenerating a projection core tensor defining a correspondence betweenthe high frequency components and an intermediate eigenspace of thefirst-quality image and a correspondence between the high frequencycomponents and the intermediate eigenspace of the second-quality image;a first sub-core tensor acquisition step of generating a first sub-coretensor corresponding to a condition specified by a first setting fromthe generated projection core tensor; a second sub-core tensoracquisition step of generating a second sub-core tensor corresponding toa condition specified by a second setting from the generated projectioncore tensor; a filtering process step of generating a low frequencycomponent control image in which high frequency components or the highfrequency components and medium frequency components of an input imageto be processed are extracted; a first subtensor projection step ofprojecting the low frequency component control image by a firstprojection computation using the eigenprojection matrix and the firstsub-core tensor to calculate a coefficient vector in the intermediateeigenspace; a second subtensor projection step of projecting thecalculated coefficient vector by a second projection computation usingthe second sub-core tensor and the eigenprojection matrix to generate aprojection image from the low frequency component control image; animage conversion step of generating a conversion image with an imagequality different from the input image; and an addition step of addingthe projection image and the conversion image.

An eighteenth aspect of the present invention provides the imageprocessing method according to any of the fifteenth to seventeenthaspects, characterized in that the projection computation includes aprojection computation using a local relationship.

A nineteenth aspect of the present invention provides a programcharacterized by causing a computer to function as: informationacquisition means for acquiring an eigenprojection matrix generated by aprojection computation from a studying image group including at leastone of an image pair formed by high frequency components of afirst-quality image and a second quality image with different imagequalities and an image pair formed by the high frequency components andmedium frequency components of the first-quality image and thesecond-quality image and acquiring a projection core tensor generatedfrom the studying image group and the eigenprojection matrix; firstsub-core tensor generation means for generating a first sub-core tensorcorresponding to a condition specified by a first setting from theacquired projection core tensor; second sub-core tensor generation meansfor generating a second sub-core tensor corresponding to a conditionspecified by a second setting from the acquired projection core tensor;filtering means for generating a low frequency component control imagein which high frequency components or the high frequency components andmedium frequency components of an input image to be processed areextracted; first subtensor projection means for projecting the lowfrequency component control image by a first projection computationusing the eigenprojection matrix and the first sub-core tensor tocalculate a coefficient vector in the intermediate eigenspace; secondsubtensor projection means for projecting the calculated coefficientvector by a second projection computation using the second sub-coretensor and the eigenprojection matrix to generate a projection imagefrom the low frequency component control image; image conversion meansfor generating a conversion image with an image quality different fromthe input image; and addition means for adding the projection image andthe conversion image.

A twentieth aspect of the present invention provides a programcharacterized by causing a computer to function as: informationacquisition means for acquiring an eigenprojection matrix generated by aprojection computation from a studying image group including at leastone of an image pair formed by high frequency components of afirst-quality image and a second quality image with different imagequalities and an image pair formed by the high frequency components andmedium frequency components of the first-quality image and thesecond-quality image, acquiring a first sub-core tensor corresponding toa condition specified by a first setting, the first sub-core tensorgenerated using a projection core tensor generated from the studyingimage group and the projection matrix, and acquiring a second sub-coretensor corresponding to a condition specified by a second setting, thesecond sub-core tensor generated using the projection core tensor;filtering means for generating a low frequency component control imagein which high frequency components or the high frequency components andmedium frequency components of an input image to be processed areextracted; first subtensor projection means for projecting the lowfrequency component control image by a first projection computationusing the eigenprojection matrix and the first sub-core tensor tocalculate a coefficient vector in the intermediate eigenspace; secondsubtensor projection means for projecting the calculated coefficientvector by a second projection computation using the second sub-coretensor and the eigenprojection matrix to generate a projection imagefrom the low frequency component control image; image conversion meansfor generating a conversion image with an image quality different fromthe input image; and addition means for adding the projection image andthe conversion image.

A twenty-first aspect of the present invention provides a programcharacterized by causing a computer to function as: eigenprojectionmatrix generation means for generating an eigenprojection matrixgenerated by a projection computation from a studying image groupincluding at least one of an image pair formed by high frequencycomponents of a first-quality image and a second quality image withdifferent image qualities and an image pair formed by the high frequencycomponents and medium frequency components of the first-quality imageand the second-quality image; projection core tensor generation meansfor generating a projection core tensor defining a correspondencebetween the high frequency components and an intermediate eigenspace orbetween the high frequency components as well as the medium frequencycomponents and the intermediate eigenspace of the first-quality imageand a correspondence between the high frequency components and theintermediate eigenspace or between the high frequency components as wellas the medium frequency components and the intermediate eigenspace ofthe second-quality image; first sub-core tensor acquisition means forgenerating a first sub-core tensor corresponding to a conditionspecified by a first setting from the generated projection core tensor;second sub-core tensor acquisition means for generating a secondsub-core tensor corresponding to a condition specified by a secondsetting from the generated projection core tensor; filtering means forgenerating a low frequency component control image in which highfrequency components or the high frequency components and mediumfrequency components of an input image to be processed are extracted;first subtensor projection means for projecting the low frequencycomponent control image by a first projection computation using theeigenprojection matrix and the first sub-core tensor to calculate acoefficient vector in the intermediate eigenspace; second subtensorprojection means for projecting the calculated coefficient vector by asecond projection computation using the second sub-core tensor and theeigenprojection matrix to generate a projection image from the lowfrequency component control image; image conversion means for generatinga conversion image with an image quality different from the input image;and addition means for adding the projection image and the conversionimage.

A twenty-second aspect of the present invention provides the programaccording to any of the nineteenth to twenty-first aspects,characterized in that the projection computation includes a projectioncomputation using a local relationship.

A twenty-third aspect of the present invention provides a dataprocessing device characterized by including: information acquisitionmeans for acquiring an eigenprojection matrix generated by a projectioncomputation from a studying data group including at least a data pairformed by medium frequency components or high frequency components offirst-condition data and second-condition data with different conditionsand acquiring a first sub-core tensor created corresponding to acondition specified by a first setting, the first sub-core tensorcreated from a projection core tensor that is generated from thestudying data group and the eigenprojection matrix and that defines acorrespondence between the first-condition data and an intermediateeigenspace and a correspondence between the second-condition data andthe intermediate eigenspace; filtering means for generating lowfrequency component control input data in which high frequencycomponents or the high frequency components and medium frequencycomponents of input data to be processed are extracted; and firstsubtensor projection means for projecting the low frequency componentcontrol input data by a first projection computation using theeigenprojection matrix and the first sub-core tensor acquired from theinformation acquisition means to calculate a coefficient vector in theintermediate eigenspace.

A twenty-fourth aspect of the present invention provides a dataprocessing device characterized by including: information acquisitionmeans for acquiring an eigenprojection matrix generated by a projectioncomputation from a studying data group including at least a data pairformed by medium frequency components or high frequency components offirst-condition data and second-condition data with different conditionsand acquiring a first sub-core tensor created corresponding to acondition specified by a first setting, the first sub-core tensorcreated from a projection core tensor that is generated from thestudying data group and the eigenprojection matrix and that defines acorrespondence between the first-condition data and an intermediateeigenspace and a correspondence between the second-condition data andthe intermediate eigenspace; filtering means for generating lowfrequency component control input data in which high frequencycomponents or the high frequency components and medium frequencycomponents of input data to be processed are extracted; and firstsubtensor projection means for projecting the low frequency componentcontrol input data by a first projection computation using theeigenprojection matrix and the first sub-core tensor acquired from theinformation acquisition means to calculate a coefficient vector in theintermediate eigenspace.

A twenty-fifth aspect of the present invention provides the dataprocessing device according to the twenty-third or twenty-fourth aspect,characterized in that the projection computation includes a projectioncomputation using a local relationship.

A twenty-sixth aspect of the present invention provides a dataprocessing method characterized by including: an information acquisitionstep of acquiring an eigenprojection matrix generated by a projectioncomputation from a studying data group including at least a data pairformed by medium frequency components or high frequency components offirst-condition data and second-condition data with different conditionsand acquiring a first sub-core tensor created corresponding to acondition specified by a first setting, the first sub-core tensorcreated from a projection core tensor that is generated from thestudying data group and the eigenprojection matrix and that defines acorrespondence between the first-condition data and an intermediateeigenspace and a correspondence between the second-condition data andthe intermediate eigenspace; a filtering step of generating lowfrequency component control input data in which high frequencycomponents or the high frequency components and medium frequencycomponents of input data to be processed are extracted; and a firstsubtensor projection step of projecting the low frequency componentcontrol input data by a first projection computation using theeigenprojection matrix and the first sub-core tensor acquired in theinformation acquisition step to calculate a coefficient vector in theintermediate eigenspace.

A twenty-seventh aspect of the present invention provides a dataprocessing method characterized by including: an information acquisitionstep of acquiring an eigenprojection matrix generated by a projectioncomputation from a studying data group including at least a data pairformed by medium frequency components or high frequency components offirst-condition data and second-condition data with different conditionsand acquiring a first sub-core tensor created corresponding to acondition specified by a first setting, the first sub-core tensorcreated from a projection core tensor that is generated from thestudying data group and the eigenprojection matrix and that defines acorrespondence between the first-condition data and an intermediateeigenspace and a correspondence between the second-condition data andthe intermediate eigenspace; a filtering step of generating lowfrequency component control input data in which high frequencycomponents or the high frequency components and medium frequencycomponents of input data to be processed are extracted; and a firstsubtensor projection step of projecting the low frequency componentcontrol input data by a first projection computation using theeigenprojection matrix and the first sub-core tensor acquired in theinformation acquisition step to calculate a coefficient vector in theintermediate eigenspace.

A twenty-eighth aspect of the present invention provides the dataprocessing method according to the twenty-sixth or twenty-seventhaspect, characterized in that the projection computation includes aprojection computation using a local relationship.

A twenty-ninth aspect of the present invention provides a programcharacterized by causing a computer to function as: informationacquisition means for acquiring an eigenprojection matrix generated by aprojection computation from a studying data group including at least adata pair formed by medium frequency components or high frequencycomponents of first-condition data and second-condition data withdifferent conditions and acquiring a first sub-core tensor createdcorresponding to a condition specified by a first setting, the firstsub-core tensor created from a projection core tensor that is generatedfrom the studying data group and the eigenprojection matrix and thatdefines a correspondence between the first-condition data and anintermediate eigenspace and a correspondence between thesecond-condition data and the intermediate eigenspace; filtering meansfor generating low frequency component control input data in which highfrequency components or the high frequency components and mediumfrequency components of input data to be processed are extracted; andfirst subtensor projection means for projecting the low frequencycomponent control input data by a first projection computation using theeigenprojection matrix and the first sub-core tensor acquired from theinformation acquisition means to calculate a coefficient vector in theintermediate eigenspace.

A thirtieth aspect of the present invention provides a programcharacterized by causing a computer to function as: informationacquisition means for acquiring an eigenprojection matrix generated by aprojection computation from a studying data group including at least adata pair formed by medium frequency components or high frequencycomponents of first-condition data and second-condition data withdifferent conditions and acquiring a first sub-core tensor createdcorresponding to a condition specified by a first setting, the firstsub-core tensor created from a projection core tensor that is generatedfrom the studying data group and the eigenprojection matrix and thatdefines a correspondence between the first-condition data and anintermediate eigenspace and a correspondence between thesecond-condition data and the intermediate eigenspace; filtering meansfor generating low frequency component control input data in which highfrequency components or the high frequency components and mediumfrequency components of input data to be processed are extracted; andfirst subtensor projection means for projecting the low frequencycomponent control input data by a first projection computation using theeigenprojection matrix and the first sub-core tensor acquired from theinformation acquisition means to calculate a coefficient vector in theintermediate eigenspace.

A thirty-first aspect of the present invention provides the programaccording to the twenty-ninth or thirtieth aspect, characterized in thatthe projection computation includes a projection computation using alocal relationship.

Regarding the twenty-third to thirty-first aspects, an example of anapplication to personal authentication based on face images will bedescribed. Although there may be a plurality of conditions (one or moreconditions in general), such as to the front, to the left, and to theright, in relation to the direction of the face in the personalauthentication based on face images, there is a property that theprojection results substantially gather into one point on a commonsecond eigenspace (i.e. “intermediate eigenspace” such as a personaldifference eigenspace) as a result of projection on the secondeigenspace from a first eigenspace (i.e. pixel eigenspace) by preservingthe locality through a modality of “direction” with one or moreconditions, regardless of the direction in the input image if the personis the same person. In this way, since projection from the firsteigenspace to the second eigenspace is possible, the condition fordetermining the positional relationship (“closeness”) between thestudying samples and the input samples does not have to be prepared foreach condition of direction (to the front, to the left, to the right, .. . ) on the second eigenspace, and one or more conditions can beaccurately handled by a single standard. Furthermore, the robustness canbe attained by suppressing specific components, such as low frequencycomponents, including a disturbance or noise. Therefore, a highlyaccurate and robust process is possible, and an advantageous effect ofspeeding up the process and controlling the memory size can be obtained.

A thirty-second aspect of the present invention provides a recordingmedium recording the program according to any of the nineteenth totwenty-second and twenty-ninth to thirty-first aspects.

The same means (steps) as in the image processing device and method andthe program according to the first to twenty-third aspects can beapplied to the means (steps), such as the filtering means (step), in thedata processing device and method and the program according to thetwenty-fourth to thirty-first aspects.

The same means or the steps corresponding to the means as in the fourthto thirteenth aspects can be added to the method inventions according tothe fifteenth to eighteenth and twenty-sixth to twenty-eighth aspects,to the program inventions according to the nineteenth to twenty-secondand twenty-ninth to thirty-first aspects, and to the inventions of thedata processing device according to the twenty-third to twenty-fifth.

In the recording medium according to the thirty-second aspect, the meanscan be added to the program recorded in the recording medium.

Advantageous Effects of Invention

According to the present invention, the image processing device thatobtains a high quality output image from a low quality input imagesuppresses the low frequency components of the input image to execute ahigh image quality formation process by the tensor projection. As aresult, the influence of image degradation in the high image qualityformation process by the tensor projection due to a disturbance ornoise, such as an illumination change, included in the low frequencycomponents can be removed from the output image, and robustness(strength) for the low frequency components (such as a disturbance andnoise) can be increased in the restored high quality image.

Limiting the target of the projection conversion to the high frequencycomponents or the medium frequency components and the high frequencycomponents, in which the low frequency components are controlled fromall frequency components, can allocate the entire eigenspace, which canbe used to generate the studying image group, to the high frequencycomponents or the medium frequency components. A highly accurate andhighly robust restored image can be obtained by fewer studying samples.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a conceptual diagram of tensor projection;

FIG. 2 is an explanatory diagram of a principle of applying the tensorprojection to super-resolution image conversion;

FIG. 3A is a block chart illustrating a summary of a process in an imageprocessing device according to an embodiment of the present invention;

FIG. 3B is a diagram illustrating frequency characteristics of an inputimage;

FIG. 3C is a diagram illustrating frequency characteristics of the inputimage after passing through a high pass filter;

FIG. 3D is a diagram illustrating frequency characteristics of an outputimage;

FIG. 4 is an explanatory diagram illustrating a change on an LPPeigenspace (personal difference eigenspace here) with a property closeto a linear shape;

FIG. 5A is an example indicating an LPP projection distribution of animage sample (low resolution) in a two-dimensional subspace;

FIG. 5B is an example indicating an LPP projection distribution of animage sample (high resolution) on a two-dimensional subspace;

FIG. 6 is a block diagram illustrating a configuration of the imageprocessing device according to the embodiment of the present invention;

FIG. 7A is a conceptual diagram of projection based on principalcomponent analysis (PCA);

FIG. 7B is a conceptual diagram of projection based on singular valuedecomposition (SVD);

FIG. 8 is a conceptual diagram illustrating an advantageous effect ofredundancy elimination by studying set representative value formation;

FIG. 9 is a diagram illustrating an example of weights defined inassociation with the distance from a concealment candidate position;

FIG. 10 is a conceptual diagram illustrating a relationship between astudying image vector group and unknown image vectors on a personaldifference eigenspace;

FIG. 11 is a diagram illustrating an example of weights defined inassociation with the distance from a studying set;

FIG. 12 is a block diagram illustrating a configuration of the imageprocessing device according to another embodiment of the presentinvention;

FIG. 13 is a configuration diagram illustrating an example of an imageprocessing system according to the embodiment of the present invention;

FIG. 14 is a block diagram illustrating an example of a configuration ofan image processing device 220 in FIG. 13;

FIG. 15 is a block diagram illustrating an example of a configuration ofa feature area specifying unit 226 in FIG. 14;

FIG. 16 is an explanatory diagram illustrating an example of a processof specifying feature areas from an image;

FIG. 17 is an explanatory diagram illustrating another example of theprocess of specifying feature areas from an image;

FIG. 18 is an explanatory diagram illustrating an example of adetermination process of feature areas by a second feature areaspecifying unit 620 in FIG. 15;

FIG. 19 is a block diagram illustrating an example of a configuration ofa compression unit 232 in FIG. 14;

FIG. 20 is a block diagram illustrating another example of aconfiguration of the compression unit 232;

FIG. 21 is a block diagram illustrating an example of a configuration ofan image processing device 250 in FIG. 13;

FIG. 22 is a block diagram illustrating an example of a configuration ofan image processing unit 330 in FIG. 21;

FIG. 23 is a diagram illustrating an example of parameters stored in aparameter storage unit 1010 in FIG. 22 in a table format;

FIG. 24 is a diagram illustrating an example of weighting of a specificparameter;

FIG. 25 is a block diagram illustrating an example of a configuration ofa display device 260 in FIG. 13;

FIG. 26 is a diagram illustrating an example of a display area of image;and

FIG. 27 is a configuration diagram illustrating an example of an imageprocessing system according to another embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present invention will be described indetail with reference to the attached drawings.

Although the present invention can be applied to various applications, aface image of a person is handled here, and an example of restoring ahigh quality image from a low quality input image will be described.

<Principle of Projection Conversion for Restoring High Quality FaceImage from Low Quality Face Image>

First, a principle of projection conversion will be described. In apreparation stage for executing a process of restoring a high qualityimage from a low quality input image, data of face images of a pluralityof people are studied in advance, and a function for defining theconversion relationship is obtained. The process will be called astudying step. A step of using the conversion function obtained in thestudying step to obtain a high quality output image from an arbitraryinput image (low image quality) will be called a restoration step.

(About Studying Image Set)

First, a studying image group including pairs of low resolution imagesand high resolution images of faces of a plurality of people (forexample, 60 people) is prepared as a studying image set. In the studyingimage set used here, images with reduced image quality obtained byreducing information under certain conditions, such as by thinning outpixels from high resolution studying images at a certain rate, are usedas low resolution studying images. The correspondence between the pairsof the low resolution studying images generated by reducing theinformation and the corresponding original high resolution studyingimages (images of the same contents of the same people) is studied inadvance to generate conversion functions (tensors defining theprojection).

A target image size (the number of pixels) and a gradation indicatingthe density are not particularly limited, for example, the number ofpixels of a high resolution image (hereinafter, may be abbreviated as an“H image”) is 64×48 pixels, the number of pixels of a low resolutionimage (hereinafter, may be abbreviated as an “L image”) is 32×24 pixels,and each pixel has 8 bits and a density value (pixel value) of 0 to 255gradation in the image data in the description.

Combining an input dimension and an output dimension can handle an inputspace and an output space in the same space (coordinate axis), and thisis convenient in the computation. In the studying step of the presentexample, the studying data of the L image is used by applying anenlargement process by an arbitrary method to conform to the number ofpixels of the H image. The correspondence of the pixels (positionalrelationship) between the L image and the H image with the matchednumber of pixels is determined one to one, and the images have the samedimension numbers and can be handled as points (coefficient vectors) inthe same coordinate space.

Images of a variety of modalities can be included in the studying imageset. To simplify the description, it is assumed here that the directionof the face is to the front and the expression of the face is standardemotionless (“normal”). In the present example, one image is dividedinto a grid area by area with a predetermined number of pixels (forexample, 8×8 pixels), and for the plurality of divided blocks(hereinafter, called “patches”), a computation process is executed patchby patch. More specifically, the number of pixels per patch×the numberof patches (the number of divisions) is the total number of processingtargets of one image.

An image with 64×48 pixels is divided by the unit (patch) of 8×8 pixelsto divide the image into 48 (8×6) patches in the description, but thepatch size, the number of divisions, the format of division, and thelike are not particularly limited. A predetermined amount of pixels canbe overlapped between adjacent patches in the division of the image, orthe process can be executed image by image without dividing the imageinto patches.

Variations of the modalities in the present embodiment using thestudying image set and dimension numbers of the modalities are puttogether in the following table (Table 1).

TABLE 1 Modality Number Modality Dimension Number No. 1 Pixel Value 8 ×8 No. 2 Resolution 2 No. 3 Patch Position 48 No. 4 Personal Difference60

The arrangement is not limited to the example of Table 1, and the numberof modalities can be further increased. In this way, various modalitiescan be added, such as the direction of the face can include ten patternswith the direction changed in ten stages in a range of “to the right-tothe front-to the left”, the expression of the face can include fourpatterns of normal, smiling, angry, and shouting expressions, and thedirection of illumination can include five patterns with the directionchanged in five stages by 45 degrees in a range of “directly horizontalto the right-to the front-directly horizontal to the left” (see Table2).

TABLE 2 Modality Number Modality Dimension Number No. 1 Pixel 8 × 8 No.2 Resolution 2 No. 3 Patch Position 48 No. 4 Personal Difference 60 No.5 Face Direction 10 No. 6 Expression 4 No. 7 Illumination Direction 5

Obviously, Tables 1 and 2 are just examples, and other modalities, suchas race, sex, and age, may be added, or the modalities may be replacedby other modalities.

The number of types of the modalities is equivalent to the order of acore tensor G defining a projection relationship described later(fourth-order tensor in the case of Table 1), and the product of thedimension numbers of the modalities is the number of components of thecore tensor G. In the case of Table 1, the number of components (size)of the core tensor G is 8×8×2×48×60.

The order of the core tensor is seven in the case of Table 2, and thenumber of components is 8×8×2×48×60×10×4×5. In this way, the order ofthe tensor increases when the modalities are added, and the number ofcomponents of the tensor rapidly increases based on the product of thedimension numbers. Therefore, it is desirable to moderately reduce thedimensions from the viewpoint of controlling the increase in the memoryand reducing the processing time (processing load reduction). Thepresent embodiment provides means that can attain high restorationcharacteristics while controlling the increase in the memory andreducing the processing time by reducing the dimensions.

(Description of Tensor Projection)

FIG. 1 is a conceptual diagram of tensor projection. Although thedescription is based on a three-dimensional space for the convenience ofthe illustration, the space can be expanded to arbitrary finitedimensions (N dimensions). The tensor projection allows transition froma real space R to an eigenspace (also called “feature space”) A andallows transition (projection) between a plurality of eigenspaces A, B,and C.

In FIG. 1, the projection relationship from the real space R to theeigenspace A is indicated by a tensor U, and the projection relationshipbetween the eigenspaces A and B is indicated by a tensor G₁ or G₁ ⁻¹.Similarly, the projection relationship between the eigenspaces B and Cis indicated by a tensor G₂ or G₂ ⁻¹, and the projection relationshipbetween the eigenspaces C and A is indicated by a tensor G₃ or G₃ ⁻¹. Inthis way, a conversion route (projection route) rotating a plurality ofeigenspaces can be designed, and data can be handled in various spaces.

FIG. 2 illustrates a principle of applying the tensor projection tosuper-resolution image conversion.

An example of FIG. 2 uses a projection between a pixel real space, apixel eigenspace, and a personal difference eigen (person feature) spaceto schematize a process of converting (reconstructing) a low resolutionimage to a high resolution image.

Numeric values (pixel values) indicating the densities are provided tothe pixels in the image data, and a coefficient vector in amultidimensional space based on an axis indicating the density value(pixel value) can be recognized at each pixel position. Considering athree-dimensional model as in FIG. 2 for the convenience of thedescription, for example, low resolution face image data of Person A isplotted as a point P_(LA) on the pixel real space. More specifically, acoefficient vector (x₁, x₂, x₃) of the low resolution face image data ofPerson A has a value (x₁) of 0 to 255 on the axis of a first basecomponent e₁, and similarly, has values (x₂) and (x₃) of 0 to 255 on theaxis of a second base component e₂ and the axis of a third basecomponent e₃, respectively, and therefore, the image data is indicatedas the point P_(LA) on the pixel real space. Similarly, high resolutionface image data of Person A is plotted as a point P_(HA) on the pixelreal space.

An object of the conversion here is to convert a point of a lowresolution image (for example, the point P_(LA) of a low resolutionimage) on the pixel real space to make a transition to a point (P_(HA)′)of a high resolution image.

In the conversion process, the points are projected from the pixel realspace R of FIG. 2( a) to the eigenspace A based on a projection functionU_(pixels) ⁻¹ using an eigenprojection matrix U_(pixels) of linearprojection by the dimension reduction method represented by localitypreserving projection (LPP) (FIG. 2( b)).

The axis (base) of the pixel eigenspace A corresponds to a feature axis(eigenvector) by the dimension reduction method, and the projection canbe recognized as a rotation of a coordinate system that converts theaxis of the pixel real space R to the axis of the pixel eigenspace A.

The points are further moved from the pixel eigenspace A to the personaldifference eigen (person feature) space B (FIG. 2( c)). A functiondefining the correspondence between the low resolution image and thepersonal difference eigenspace is used as a projection function G_(L)⁻¹. As illustrated in FIG. 2( c), the point of the low resolution imageand the point of the high resolution image of the same person can beplotted at substantially the same position in the personal differenceeigenspace. Using this property, a projection function G_(H) definingthe correspondence between the high resolution image and the personaldifference eigenspace is used to restore the pixel eigenspace A from thepersonal difference eigenspace.

As illustrated in FIG. 2( d), after restoring the pixel eigenspace A byG_(H) that is a different function from G_(L), the pixel real space A isfurther restored based on a projection function U_(pixels) using theeigenprojection matrix (FIG. 2( e)). In this way, the substantialconformity between the L image point and the H image point in thepersonal difference space can be used to convert the L image to the Himage through the route of (c)→(d)→(e) of FIG. 2.

More specifically, assuming that V is a personal difference eigenspacecoefficient vector in the personal difference eigenspace of FIG. 2( c),a high resolution pixel vector H in the pixel real space can be obtainedby the following formula.H=U _(pixels) G _(H) V  [Expression 1]

Meanwhile, a low resolution pixel vector L in the pixel real space isalso as in the following formula.L=U _(pixels) G _(L) V  [Expression 2]

Therefore, to reconstruct the pixel eigenspace→the pixel real spacethrough the pixel eigenspace→the personal difference eigenspace from thelow resolution image (low resolution pixel vector L) of the pixel realspace to obtain the high resolution image in the pixel real space, theconversion is possible based on the projection of the following formula.H=U _(pixels) G _(H) V=U _(pixels) G _(H)(U _(pixels) G _(L))⁻¹L  [Expression 3]

In the present embodiment, the projection function (U_(pixels)) isobtained from the studying image set including a pair group of lowresolution images and high resolution images using the localitypreserving projection (LPP), and based on this, the projection functionsG_(L) and G_(H) are obtained so that the L image point and the H imagepoint of the same person substantially match on the personal differencespace.

A low resolution image can be accurately converted to a high resolutionimage based on the obtained projection functions (U_(pixels), G_(L),G_(H)) and the framework of the projection route illustrated in FIG. 2.

Although an example of the LPP projection is described in the presentembodiment, another projection method, such as principal componentanalysis (PCA), can be applied in place of the LPP projection to carryout the present invention.

<Outline of LPP Projection>

The following is an outline of a computation procedure of the LPPprojection.

(Procedure 1): obtain a similarity matrix: S indicating whether thestudying samples (round-robin) are similar.

(Procedure 2): obtain Σ of each line of the similarity matrix S toobtain a diagonal matrix: D.

(Procedure 3): obtain a Laplacian matrix: L=D−S.

(Procedure 4): solve the following general eigenvalue problem.X·L·X ^(T) ·u=λ·X·D·X ^(T) ·u  [Expression 3]

For example, [1] Cholesky decomposition or [2] general eigenvalueproblem is converted to an eigenvalue problem based on inverse matrixcalculation to solve the problem.

(Procedure 5): sort eigenvectors u corresponding to eigenvalues in theascending order of eigenvalues λ to obtain an LPP projection matrix: U.

<Summary of Process>

FIG. 3A is a block chart illustrating a summary of a process in anembodiment of the present invention. As illustrated, the processaccording to the present embodiment can be roughly divided into astudying step and a restoration step.

In the studying step, a studying image group (input studying image set)including pairs of low quality images and high quality images is input(#10), and a high pass filter (high-pass filter) is used for the imagegroup to extract high frequency components of the studying image set(low quality images and high quality images) (#11). A process ofapplying a dimension reduction method, such as locality preservingprojection (LPP), to the high frequency components of the input imagesto generate a projection tensor is executed (#12).

It is sufficient if a “high frequency component extraction step”illustrated by providing #11 in FIG. 3A at least controls the lowfrequency components including an illumination change factor, and mediumfrequency components may be extracted with the high frequencycomponents. More specifically, the high frequency components or the highfrequency components and the medium frequency components of the inputstudying image set are extracted, and the studying image set withcontrolled low frequency components is obtained.

In the projection tensor generation step (#12), an eigenprojectionmatrix (#14) is generated, and a projection core tensor (#16) definingthe correspondence between the low quality images and an intermediateeigenspace and the correspondence between the high quality images andthe intermediate eigenspace is generated.

Describing an example of the LPP projection, the LPP converts thecoordinates to preserve the closeness of local values of samples(information of geometric distance of neighborhood values) in theoriginal space (real space of pixels here), and the coordinate axis isdetermined to embed neighborhood samples in the original space to theneighborhood in the projected space (eigenspace).

For example, in the studying image set of Table 1, H images and L imagesof 60 people are plotted in the real space of pixel at each patchposition, and the LPP is applied to the distribution of 120 points toobtain a feature axis focusing on the points with close values (pointswith similar changes) in the distribution.

In this way, an LPP eigenprojection matrix U_(j)={U₁, U₂, U₃, . . . U₆₄}corresponding to the dimensions (64 dimensions in the case of Table 1)of the patch positions is obtained.

The LPP eigenprojection matrix is used to generate an LPP projectioncore tensor G including the correspondence between the L images and thepersonal difference eigenspace (tensor GL_(j)={GL₁, GL₂, GL₃, . . .GL₆₄}) and the correspondence between the H images and the personaldifference eigenspace (tensor GH_(j)={GH₁, GH₂, GH₃, . . . GH₆₄}).

More specifically, the eigenprojection matrix U is obtained from theviewpoint of the modalities, such as pixels, resolutions, and patchpositions, and the U is used to obtain projection core tensor Gcomponents, and the set of the components is obtained as the projectioncore tensor G.

In the LPP, the alignment (arrangement) of the feature axes isdetermined in ascending order of the eigenvalues. Therefore, thedimensions are reduced by using only highly influential higher-levelfeature axes, and the size of the core tensor can be significantlyreduced.

In the process of the calculation, all eigenprojection matrices Uincluding less influential matrices are calculated. When the matricesare actually used in the restoration process, less influential matricesare not used, and some of the highly influential matrices can be usedfor the restoration. The execution of moderate dimension compression ofthe feature axes can set the size of the projection core tensor to anappropriate size.

Meanwhile, in the restoration step, a low quality image as a conversionsource is input (#20), and information for specifying the patch positionto be processed and information for setting a distinction between the Limage and the H image are provided (#22).

A first sub-core tensor (GL_(j)={GL₁, GL₂, GL₃, . . . GL₆₄} in the aboveexample of Table 1) corresponding to an L setting as a first setting isgenerated (#24) from the projection core tensor G (#16) generated in thestudying step, and a second sub-core tensor (GH_(j)={GH₁, GH₂, GH₃, . .. GH₆₄} in the above example of Table 1) corresponding to an H settingas a second setting is generated (#26).

The projection core tensor (#16) is created based on all eigenvectorscorresponding to the modalities and is a set including the projectioncomponents related to all modalities. Therefore, components to be usedin the restoration process need to be extracted from the tensorcomponents. For example, a condition of using an eigenspace of “personaldifference” as an intermediate eigenspace in the projection routedescribed in FIG. 2 (space at a turning point of the projection route)can be determined to extract sub-core tensors GL and GH corresponding tothe condition. In this way, the step to the generation of the sub-coretensors to be actually used may be included in the “studying step”.

A high frequency component extraction process using a high pass filteris applied (#21) to the inputted low quality image (#20). In the highfrequency component extraction step, the same process as the highfrequency component extraction step (#11) in the studying step isexecuted. For example, a process of extracting, from the input image,the same frequency components as the frequency components extracted fromthe studying image set is executed. More specifically, in the highfrequency component extraction step of the restoration step, the samefrequency components as the studying image set that serves as a basis ofthe eigenprojection matrix and the projection core tensor are extracted.

Characteristics illustrated by providing reference numeral 20 in FIG. 3Billustrate a relationship (frequency characteristics of input image)between a spatial frequency (frequency) in the input image and aresponse (gain). As illustrated in FIG. 3B, the input image includes aspatial frequency up to f₂, and a low frequency area (for example,frequency area below f₁) includes an illumination change factor.

Characteristics illustrated by providing reference numeral 21 in FIG. 3Care frequency characteristics of a low frequency component control imagein which the high frequency components are extracted from the inputimage (#20 of FIG. 3A). A process of cutting the frequency componentsbelow f₁ is applied to the input image including the frequencycharacteristics illustrated in FIG. 3B.

When the low frequency component control image including the frequencycharacteristics illustrated by providing reference numeral 21 to FIG. 3Cis generated, the low frequency component control image is projectedusing the eigenprojection matrix and the first sub-core tensor (#30) tocalculate an intermediate eigenspace coefficient vector. The firstsubtensor projection step (#30) is equivalent to the projection of theroute described in (a)→(b)→(c) of FIG. 2.

The second sub-core tensor and the eigenprojection matrix are used toproject the image (#34) to obtain a projection image for the input imagein which the low frequency components are controlled. The secondsubtensor projection step (#34) is equivalent to the projection of theroute described in (c)→(d)→(e) of FIG. 2.

Meanwhile, although not illustrated in FIG. 3A, an enlargement processfor enlarging to the same size (the number of pixels) as the highquality image (#36) is applied to the low quality image (input image,#20) to generate an enlarged image. The frequency characteristics of theenlarged image are as illustrated by providing reference numeral 21′ inFIG. 3D.

In an addition step illustrated by providing reference numeral #60 inFIG. 3A, a process of adding the enlarged image and the projection imagegenerated by the tensor projection is executed, and a restored image(high quality image, #36) is generated by adding, to the enlarged image,the projection image in which the image quality of the high frequencycomponents of the input image are increased by the tensor projection.

FIG. 3D illustrates an example of the frequency characteristics of thehigh quality image illustrated by providing reference numeral #36 inFIG. 3A. The characteristics illustrated by providing reference numeral20′ in FIG. 3D are frequency characteristics of the enlarged image, andthe characteristics illustrated by providing reference numeral 35 arefrequency characteristics of the projection image. If thecharacteristics are added, an output image (high quality image, #36)including frequency characteristics illustrated by a solid line can beobtained.

As illustrated in FIG. 3D, although the response of the frequency areagreater than f₁ decreases (restoration characteristics are degraded) inthe enlarged image enlarged from the input image (20′), a predeterminedresponse (restoration characteristics) is secured for the frequency areafrom f₂ to f₂′ by adding the projection image (35). Therefore, accordingto the image processing illustrated in the present example, thefrequency area from f₂ to f₂′ not expressed in the input image can beexpressed in the restored output image.

In FIG. 3D, f₁′ denotes a frequency corresponding to a threshold f₁ inthe input image, and there is a method of setting the frequency f₁′based on the Nyquist frequency in a sampling theorem. More specifically,using the frequency f₁ corresponding to a frequency little lower thanthe Nyquist frequency as a threshold to apply a high frequency componentextraction process to the input image can remove the image qualitydegradation factor included in the low frequency components of the inputimage, and a preferable high quality image is restored.

The frequency area extracted in the input image (and the studying imageset) may be a so-called cutoff frequency (frequency in which theresponse is −3 dB) or may be arbitrarily set according to the inputimage and the output image.

In the addition step (#60) illustrated in FIG. 3A, it is also preferableto add the enlarged image and the projection image after weighting theenlarged image and the projection image using the weighting factorsdetermined from the reliability of the projection image as an index.

For example, the projection image can be actively used when therestoration reliability of the high image quality formation process bythe tensor projection is high, and the weighting factors can bedetermined to increase the rate of adopting the enlarged image when therestoration reliability is low. It is more preferable to take thefrequency characteristics into account to determine the weightingfactors.

Although not illustrated in FIG. 3A, it is also preferable to includestorage means for storing the eigenprojection matrix (#14) and theprojection core tensor (#16) generated and acquired in the studyingstep. The storage means may be a semiconductor storage element, such asa memory, or various storage media (elements), such as a magneticstorage medium like an HDD and an optical storage medium, can beapplied. The storage means may be embedded in the device or may beremovable from the device, such as a memory card.

The step of generating the projection tensor (#12) in FIG. 3A and thecomputation means are equivalent to “eigenprojection matrix generationmeans (step)” and “projection core tensor creation means (step)”. Thestep of generating the first sub-core tensor (#24) and the computationmeans are equivalent to “first sub-core tensor creation means (step)”,and the step of generating the second sub-core tensor (#26) and thecomputation means are equivalent to “second sub-core tensor creationmeans (step)”.

The low quality image (#20) as a conversion source is equivalent to an“input image”, and the high frequency component extraction step (#21) bythe high pass filter is equivalent to “filtering means (step)”.

The step of the second subtensor projection (#30) and the computationmeans are equivalent to “second subtensor projection means (step)”, andthe projection image of the high frequency components obtained in thesecond subtensor projection (#34) is equivalent to a “projection image”.

The addition step of adding the enlarged image and the projection image(#60) is equivalent to “addition means (step)”.

In the present example, although the image processing for removing theimage quality degradation factor of the restoration image due to theillumination change included in the low frequency components in theinput image and the output image is described, the image processingmethod can be applied to factors other than the illumination change.

For example, a medium frequency area of the input image can becontrolled for an image quality degradation factor included in themedium frequency area, and a high image quality formation process (forexample, enlargement process) based on a system different from thetensor projection can be used for the medium frequency area. High imagequality processing based on the tensor projection system can be used forother frequency areas, and two images generated by the high imagequality processing can be added. In this way, an image qualitydegradation factor in a predetermined frequency area can be removed fromthe output image.

<About Advantage of Using LPP Projection>

FIG. 4 illustrates an example in which a change in the modality(personal difference here) on the LPP eigenspace has a property close toa linear shape. For example, when the studying images of four people,Person A, Person B, Person C, and Person D, are converted by the LPP,the change (change in the personal difference) from Person A to Person Bof FIG. 4 is a substantially smooth (continuous) change on the personaldifference eigenspace and is close to a linear shape while maintainingthe local structure.

In this way, the conversion on the eigenspace of LPP high order singularvalue decomposition (LPP_HOSVD) (n=2, 3, 4 . . . ) can approximate thechange in the element of the modality corresponding to the eigenspace toa linear shape (see FIG. 4), and an arbitrary input image vector isexpressed as a highly linear interpolation point relative to the vectorgroup of the studying image sample.

More specifically, an unknown input image other than the studying imagesample can be excellently and approximately expressed on the LPPeigenspace using the vector group of the studying image samples. Thispoint is one of the advantages of using the LPP projection conversionsystem (advantage 1).

FIG. 5A indicates an LPP projection distribution of a low resolutionimage sample on a two-dimensional subspace, and FIG. 5B indicates an LPPprojection distribution of a high resolution image sample on atwo-dimensional subspace (reference: ZHUANG Yueting, ZHANG Jian, WU Fei,“Hallucinating faces: LPH super-resolution and neighbor reconstructionfor residue compensation”, Pattern Recogn, Vol. 40, No. 11, Page.3178-3194 (2007)).

As illustrated in the distributions, it is known that the topology ofthe low resolution distribution (FIG. 5A) and the topology of the highresolution distribution (FIG. 5B) of the studying image sample vectorgroup on the LPP eigenspace are highly correlated even if theeigenspaces are separately studied and converted.

Using such a property of the LPP to further express the mutualprojection relationship between both elements (low resolution and highresolution) of the modality by tensors (G_(L), G_(H)) of a framework ofmultilinear projection generates a new advantageous effect (advantage 2)that a highly accurate conversion is possible (error can be reduced).

A synergistic advantageous effect of the advantages 1 and 2 generates anew advantageous effect of further improving the accuracy of theprojection relationship. Input conditions are alleviated compared to theconventional techniques, and robustness (strength) is attained(advantage 3).

The conversion on the eigenspace based on the LPP_HOSVD (n=2, 3, 4 . . .) increases the correlation between the distributions of the studyingimage group and further reduces the dimensions of the orders(modalities). The process can be speeded up, and the memory can be saved(advantage 4).

Example of Configuration of Specific Embodiment

A further practical embodiment including the procedure of the processdescribed in FIG. 3A will be described below.

FIG. 6 is a block diagram illustrating a configuration of an imageprocessing device 100 according to an embodiment of the presentinvention. FIG. 6 is divided into the studying step and the restorationstep to clarify the correspondence with FIG. 3A, and blocks of theprocessing units contributing to the processes of the steps areillustrated in accordance with the flow of the processes.

As illustrated in FIG. 6, the image processing device 100 includes a lowresolution enlargement processing unit 102, a high pass filter 104, apatch dividing unit 106, an LPP projection tensor generation unit 108, anumber of studying representatives acquisition unit 110, a studying setrepresentative value formation processing unit 112, a re-projectiontensor generation unit 114, a set value acquisition unit 120, a firstsub-core tensor generation unit 122, a second sub-core tensor generationunit 124, a first LPP_HOSVD projection processing unit 130, acoefficient vector correction processing unit 140, a second LPP_HOSVDprojection processing unit 150, an addition unit 160, a weightcalculation unit 162, a general-purpose super-resolution processing unit164, and a combining unit 166. The means for executing the processes ofthe processing units are realized by dedicated electronic circuits(hardware), software, or a combination of the circuits and the software.

The first LPP_HOSVD projection processing unit 130 is means for carryingout the process of the projection route described in FIG. 2( a)→(b)→(c),and as illustrated in FIG. 6, includes an “L pixel→eigenspace projectionunit 132” that projects the L image from the pixel real space to thepixel eigenspace and an “[L pixel→personal difference] eigenspaceprojection unit 134” that projects the L image from the pixel eigenspaceto the personal difference eigenspace. A pixel value in the L image willbe called an L pixel, and a pixel value in the H image will be called anH pixel.

The second LPP_HOSVD projection processing unit 150 is means forcarrying out the process of the projection route of FIG. 2( c)→(d)→(e)and includes a “[personal difference→H pixel] eigenspace projection unit152” that projects the H image from the personal difference eigenspaceto the pixel eigenspace and an “eigenspace→H pixel projection unit 154”that performs projection from the pixel eigenspace to the real space.

Contents of the processing units of FIG. 6 will be described below.

(Low Resolution Enlargement Processing Unit)

The low resolution enlargement processing unit 102 executes a process ofenlarging an input low resolution image to a predetermined size. Theenlargement method is not particularly limited, and various methods,such as bicubic, B spline, bilinear, and nearest neighbor, can be used.

In the studying step, the input low resolution image of the studyingimage set is enlarged to the number of pixels of the same size as thehigh resolution image. In the restoration step, the input low resolutionimage is enlarged to the number of pixels of the same size as the output(in the present example, the same size as the high resolution image ofthe studying image set). This is to, as described, match the dimensionnumbers of the input and the output.

(High Pass Filter)

The high pass filter 104 applies filtering for controlling the low-passto the input image. An unsharp mask, Laplacian, gradient, or the likecan be used as the filter. Much of the influence of the illuminationchange in the face image is in a low frequency range. Therefore, theinfluence of the illumination change can be removed by suppressing thelow-pass by the high pass filter 104, and the robustness to theillumination change can be improved.

The entire eigenspace that can be used in studying can be allocated tothe high frequency components by removing the low frequency componentsfrom the input image and limiting the processing target of theprojection conversion to the high frequency components from the entirefrequency range. In the present embodiment that attempts to restore thehigh resolution output image from the low resolution input image, it isimportant to mainly restore the high frequency components. In theembodiment of the present invention that applies the tensor projectionincluding the framework of the multilinear projection described in FIG.2 to the restoration of the high frequency components, providing onlythe high frequency information as a target of the projection process canobtain a new advantageous effect of simultaneously attaining anadvantageous effect that the target can be effectively allocated to theeigenspace (entire dynamic range can be used for processing of the highfrequency components in the eigenspace) and an advantageous effect thatthe influence of the illumination change of the input image can becontrolled.

If a modality of “direction of illumination” (illumination change) isadded to study a necessary studying image group, a restoration processcorresponding to the illumination change is possible based on the sameconversion principle as in FIG. 1. However, the size of the tensorincreases, and the computation load and the memory capacity increase.

According to the configuration of using the high pass filter 104 as inthe present embodiment, there is no addition of the illumination changemodality (=no increase in the order of the tensor), and the illuminationcondition detection process is not necessary. Data collection andprocessing for studying the restoration projection based on theillumination change are not necessary. Therefore, there is an advantagethat the increase in the memory capacity can be prevented, and theprocessing load does not significantly increase.

According to the present embodiment, highly accurate and highly robustrestoration with fewer studying samples can be expected from asynergetic effect of the high pass filter 104 and the LPP_HOSVDprojection. As described, although the process of extracting the highfrequency components (frequency components having f₁ or more of FIGS. 3Bto 3D) is illustrated in the present example as an example ofcontrolling the low frequency components including the illuminationchange factor, the medium frequency components may be extracted with thehigh frequency components.

(Patch Dividing Unit)

The patch dividing unit 106 divides the inputted image into a grid. Theimage processing is executed patch by patch in the studying step and therestoration step. The execution of the process patch by patch to limitthe processing target to a part of the image can handle the projectiontarget in a low dimension, and high image quality and robustness for thechange in the personal difference can be attained. Therefore, theconfiguration including the means for the patch division is a preferablemode in the implementation of the present invention.

(LPP Projection Tensor Generation Unit)

The LPP projection tensor generation unit 108 applies the localitypreserving projection (LPP) to generate an LPP projection tensor fromthe input studying image set (pair groups of the low resolution imagesand the high resolution images) for which preprocessing, such as the lowresolution enlargement, the high pass filtering, and the patch division,is finished.

The LPP converts the coordinates to preserve the local closeness of asample (information of a geometric distance of a neighborhood) in theoriginal linear space (the real space of pixels here) and determines thecoordinate axis to embed a neighborhood sample in the original space tothe neighborhood in the projected space (eigenspace).

More specifically, when the preprocessed input studying image set isprovided, the LPP first generates the LPP eigenprojection matrixU_(pixels) based on the set and then generates the LPP projection coretensor G as in the singular value decomposition (SVD).

More specifically, a matrix M indicating the images of the studyingimage set is decomposed to M=U₁ΣU₂. Since the matrices U₁ and U₂ arealready obtained as the LPP eigenprojection matrices, Σ(=G) is obtainedfrom the matrix computation.

In the principle of “LPP locality preserving projection”, an axis(feature axis), in which the samples with similar values are close, isobtained. As a result, the local structure is preserved, and thedistance between neighborhood sample values is used. A similarity thatis large between samples (specimens) with close values and that is smallbetween samples with different values is implemented, and the projectionfor approximating the samples with large similarity is performed. TheLPP is used to hold the local closeness to reduce the linear dimensions,and the LPP has features of preserving the local geometry and being ableto perform simple projection only by the linear conversion. However, theLPP does not have an orthogonal basis in general. However, an orthogonalLPP is also proposed, and it is desirable to use the orthogonal LPP.

<About Calculation of Orthogonal LPP>

Based on the premise that a diagonal matrix D and a Laplacian matrix Lare obtained from the LPP algorithm, an orthogonal LPP projection matrixW_(OLPP)={u₁, . . . , u_(r)} is obtained by the following procedure. Adimensional number r is a number smaller than the original dimensionnumber n.

(Step 1): an eigenvector corresponding to a minimum eigenvalue of amatrix (XDX^(t))⁻¹XLX^(t) is set as u₁.

(Step 2): A k-th eigenvector is obtained. More specifically, aneigenvector corresponding to a minimum eigenvalue of a matrix M^((k))illustrated in [Expression 4] is set as u_(k).M ^((k)) ={I−(XDX ^(t))⁻¹ A ^((k-1)) [B ^((k-1))]⁻¹ [A ^((k-1))]}(XDX^(t))⁻¹(XLX ^(t))  [Expression 4]wherein

A^((k-1))={u₁, . . . , u_(k-1)}

B^((k-1))=[A^((k-1))]^(t)(XDX^(t))⁻¹A^((k-1))

The computation of step 2 is repeated from K=2 to r (to n if thedimensions are not compressed and to r if the dimensions are compressed)to obtain eigenvectors. In this way, the orthogonal LPP projectionmatrix W_(OLPP)={u₁, . . . , U_(r)} is obtained.

<Comparison with Principal Component Analysis (PCA)>

Compared to the LPP, the principle of the principal component analysis(PCA) is to maximize a global dispersion, and a main object is to hold aglobal distribution to reduce the linear dimensions. The PCA hasfeatures of preserving a global geometry and performing a simpleprojection based only on the linear conversion and has an orthogonalbasis.

As illustrated in FIG. 7A, the PCA only provides a projection functionbetween the real space vector and the eigen (feature) space vector.Meanwhile, as illustrated in FIG. 7B, the singular value decomposition(SVD) also provides a projection function Σ between the vector of theeigenspace A and the vector of the eigenspace B in addition to theprojection function U between the real space vector and the eigen(feature) space vector. Therefore, the SVD is equivalent to adecomposition expression of the feature vector in the PCA.

A matrix SVD is a method of decomposing an arbitrary matrix M intoM=UΣV*. Here, U denotes an output normal orthogonal vector, V denotes aninput normal orthogonal vector, Σ denotes a diagonal output matrix ofσi, and V* denotes an adjoint matrix of V. More specifically, a Vprojection eigenspace and a U projection eigenspace are uniquely andlinearly associated in a relationship of σi (>0) times per i. Thedimensions of the matrix SVD are increased (the modalities areincreased), that is, the tensor is formed, to obtain a tensor SVD(TSVD). The technique described in Non-Patent Literature 1 uses theTSVD.

Meanwhile, the dimensions of the LPP are increased (the modalities areincreased) in the LPP_HOSVD (n=2, 3, 4 . . . ) of the presentembodiment, and the LPP_HOSVD (n=2, 3, 4 . . . ) is a tensor version ofthe LPP. Describing an example of the studying image set of Table 1, theH images and the L images of 60 people are plotted at the patchpositions on the pixel real space, and the LPP is applied to thedistribution of the 120 points to obtain a feature axis focusing onclose values (points with similar changes) in the distribution.

However, in the present embodiment, a studying image set including pairgroups of low quality images and high quality images of more than 60people (for example, 200 people) is used in the initial studying stagefrom the viewpoint of selecting more appropriate 60 people to ultimatelydetermine the projection function from the samples of 60 people.

In this way, a provisional, tentative LPP eigenprojection matrixU_(j)={U₁, U₂, U₃, . . . U₂₀₀} corresponding to the dimensions (200dimensions in the case of Table 1) of the patch positions is obtained.The tentative LPP eigenprojection matrix U_(j) is used to generate thetentative projection core tensor G that defines the conversion betweenthe pixel eigenspace and the personal difference eigenspace for the Limages and the H images based on the tensor singular valuedecomposition.

The tentative projection core tensor G includes a sub-core tensorG_(Lj)={G_(L1), G_(L2), G_(L3), . . . G_(L200)} that associates thepixels (L pixels) of the low resolution image with the personaldifference eigenspace and a sub-core tensor G_(Hj)={G_(H1), G_(H2),G_(H3), . . . G_(H200)} that associates the pixels (H pixels) of thehigh resolution image with the personal difference eigenspace.

(Number of Studying Representatives Acquisition Unit)

As described, the studying images are narrowed down in the presentembodiment to select appropriate samples to determine the projectionfunction. The number of pair groups of the studying images (the numberof people of the samples here) to be ultimately used is called the“number of studying representatives”, and information of the number ofstudying representatives is acquired from the outside.

The number of studying representatives acquisition unit 110 of FIG. 6 ismeans for importing the number of studying representatives from theoutside.

(Studying Set Representative Value Formation Processing Unit)

The studying set representative value formation processing unit 112executes a process of obtaining a personal difference eigenspacecoefficient vector group from the preprocessed input studying image set(at least one of the low resolution images and the high resolutionimages). In the process, the same process as the first LPP_HOSVDprojection processing unit 130 in the restoration step, that is, aprocess up to L pixel→eigenspace projection (process of referencenumeral 132) and [L pixel→personal difference] eigenspace projection(process of reference numeral 134), is applied to the input studyingimage set to obtain the coefficient vector of the personal differenceeigenspace.

This is equivalent to obtaining the projection points for the personaldifference eigenspace in relation to the images of the input studyingimage set. As a result, the closeness between the sample (specimen)points in the personal difference eigenspace can be recognized.

Based on the distribution of the points in the personal differenceeigenspace, N representative personal difference eigenspace coefficientvectors (representative vectors) are obtained according to the number ofstudying representatives N obtained from the number of studyingrepresentatives acquisition unit 110. The representative vectors areobtained using a K-means method, an EM algorithm, a variational Bayesmethod, a Markov chain Monte Carlo method, or the like. Or a pluralityof the systems may be combined. For example, initial candidates can beobtained by the k-means method, and the representative vectors can beultimately obtained by the EM algorithm to highly accurately obtain therepresentative vectors in a relatively short time.

As a result of the representative value formation, similar sample points(closely located points in the personal difference eigenspace) are puttogether (replaced) to the representative vectors. Although the obtainedrepresentative vector group on the personal difference eigenspace can beused as it is, a mode of adopting N samples of the closest preprocessedinput studying image set for each of the obtained vectors of therepresentative vector group is preferable. While the representativevectors are combined from the sample points in the former case, theactual sample points are adopted in the latter case. Therefore, a blurcaused by combining the representative points can be prevented.

As a result of the representative value formation, similar sample points(closely located points in the personal difference eigenspace) arerepresented by the representative values, and the redundancy of thestudying image set is reduced.

(Re-Projection Tensor Generation Unit)

The re-projection tensor generation unit 114 applies the same process asthe LPP projection tensor generation unit 108 to the N representativestudying image sets obtained in the studying set representative valueformation processing unit 112 to generate the LPP eigenprojection matrixand the LPP projection core tensor again. In this way, an LPPeigenprojection matrix (U_(pixels)) 115 and an LPP projection coretensor (G) 116 used in the restoration step described later are obtainedbased on the representative studying image sets.

Although the LPP projection tensor generation unit 108 and there-projection tensor generation unit 114 are illustrated by separateblocks in FIG. 6, a configuration of using the same processing block toloop the process is also possible.

FIG. 8 is a conceptual diagram schematically illustrating a state ofeliminating the redundancy of the studying sets by the process offorming the studying set representative value. To simplify thedescription, the number of studying samples is “5”, and atwo-dimensional space is illustrated here. As a result of the process(first time) by the LPP projection tensor generation unit 108, when theface image data of five people of Person A to Person E is distributed asillustrated in FIG. 8 on the personal difference eigenspace, therepresentative value is formed by Person C for the samples of threepeople of Person A, Person C, and Person D in a relatively closepositional relationship, and the samples of Person A and Person D areeliminated.

Based on the data of three people of Person B, Person C, and Person E,the re-projection tensor generation unit 114 recalculates the LPPeigenprojection matrix U_(pixels) and the LPP projection core tensor G.In this way, the redundancy of the studying image sets is reduced by theprocess of forming the studying set representative value, and thedimensions of the orders of the projection tensor can be reduced whilemaintaining the restoration performance and the robustness. This cancontribute to the control of the increase in the memory and to theincrease in the speed of the process.

The processing units operated in the restoration step will be described.

The low resolution enlargement processing unit 102, the high pass filter104, and the patch dividing unit 106 described in the studying step ofFIG. 6 are also used in the same way for the input image (low qualityimage) in the restoration step. More specifically, in the restorationstep, “L pixel→eigenspace projection” (reference numeral 132), “[Lpixel→personal difference] eigenspace projection” (reference numeral134), “[personal difference→H pixel] eigenspace projection” (referencenumeral 152), and “eigenspace→H pixel projection” (reference numeral154) are performed patch by patch for the high pass components of theinput image.

(Set Value Acquisition Unit)

The set value acquisition unit 120 is means for acquiring, from theoutside, information of the patch position to be processed andinformation for designating the setting of L and H and for providing theinformation to the “first sub-core tensor generation unit 122”, the“second sub-core tensor generation unit 124”, the “L pixel→eigenspaceprojection unit 132”, and the “eigenspace→H pixel projection unit 154”.

Instead of acquiring the information from the outside, the patchpositions of the image after the patch division may be provided to the“first sub-core tensor generation unit 122”, the “second sub-core tensorgeneration unit 124”, the “L pixel→eigenspace projection unit 132”, andthe “eigenspace→H pixel projection unit 154” in association with thefirst sub-core tensor generation unit 122 and the second sub-core tensorgeneration unit 124.

The means may be performed in the studying step along with the “firstsub-core tensor generation unit 122” and the “second sub-core tensorgeneration unit 124”.

(First Sub-Core Tensor Generation Unit)

The first sub-core tensor generation unit 122 provides the patchpositions and the conditions of the L setting output from the set valueacquisition unit 120 to generate a sub-core tensor G_(L) for lowresolution from the LPP projection core tensor 116 related to the outputof the re-projection tensor generation unit 114. The means may beperformed in the studying step, and in place of or along with the modeof storing and preserving the LPP projection core tensor 116, thesub-core tensor G_(L) may be generated, stored, and preserved in thestudying step. Although a memory for preserving the sub-core tensor isnecessary according to the mode, there is an advantage that theprocessing time of the restoration step can be reduced.

(L Pixel→Eigenspace Projection Unit)

The “L pixel→eigenspace projection unit 132” in the first LPP_HOSVDprojection processing unit 130 obtains the LPP eigenprojection matrix115 (U_(pixels)) based on the patch positions provided from the setvalue acquisition unit 120 and applies, to the image of the output fromthe patch dividing unit 106, the process of U_(pixels) ⁻¹ projection tothe pixel eigenspace described in (a)→(b) of FIG. 2. By the way,U_(pixels) ⁻¹ denotes an inverse matrix of U_(pixels).

([L Pixel→Personal Difference] Eigenspace Projection Unit)

The [L pixel→personal difference] eigenspace projection unit 134following the “L pixel→eigenspace projection unit 132” in FIG. 6 obtainsa corresponding projection tensor G_(L) from the first sub-core tensorgeneration unit 122 and applies, to the output of the “Lpixel→eigenspace projection unit 132”, the process of G_(L) ⁻¹projection to the personal difference eigenspace described in (b)→(c) ofFIG. 2 to obtain the personal difference eigenspace coefficient vector.

(Coefficient Vector Correction Processing Unit)

The coefficient vector correction processing unit 140 uses the personaldifference eigenspace coefficient vector group of the number of patchesobtained by the [L pixel→personal difference] eigenspace projection unit134 of FIG. 6 to generate a correction coefficient vector group to beprovided to the [personal difference→H pixel] eigenspace projection unit152 of the second LPP_HOSVD projection processing unit 150.

In the correction computation, the features of the tensor projectionincluding a framework of multilinear projection are used. Morespecifically, when the studied LPP eigenprojection matrix and LPPprojection core tensor are used as the features of the tensor projectionas described in FIG. 2, the pixel vectors of the patch group obtained bydividing the face image of the same person (for example, face image ofPerson A) are substantially gathered into one point on the personaldifference eigenspace. Therefore, a high mutual correlation betweenpatches can be used based on the conversion on the same order of thetensor space.

The use of the property can determine the presence/absence of partialconcealment in the face image (situation that part of the face is hiddenby glasses, a mask, an edge of an automatic door, a door, and the like)and can prevent the deterioration in the restoration due to the partialconcealment. Hereinafter, some specific examples will be described.

Example of Restoration as Face in which Concealing Matter is Removedfrom Concealed Area of Face

The pixel vector of the patch with the concealing matter is a point at aposition away from the area where the pixel vectors of other patcheswithout the concealing matter gather in the personal differenceeigenspace. In such a case, the pixel vector of the patch with theconcealing matter can be corrected to a vector without the concealingmatter (correction coefficient vector).

Example A-1-1

Noise of the personal difference eigenspace coefficient vector group(influence of partially concealing matters, such as glasses, a mask, anda door) is removed using a representative value, such as an averagevalue, a median, a maximum value, and a minimum value, of thecoefficient vector group of the patch group related to the same personin the personal difference eigenspace as a value of the correctioncoefficient vector group.

Example A-1-2

The noise may be further removed using mainly a representative value,such as an average value, a median, a maximum value, and a minimumvalue, in a histogram of the coefficient vector group of the patch groupin relation to the same person in the personal difference eigenspace, orfor example, an average value, a median, a maximum value, or a minimumvalue targeting the personal difference eigenspace coefficient vectorgroup of a range of dispersion a or a range of 2σ, as a value of thecorrection coefficient vector group.

Example of Restoration for Concealing Matter (Such as Glasses and Mask)by Detection of Concealed Area

When an area with a concealing matter is detected, a mode of convertingthe area by a dedicated tensor is also possible.

Example A-2-1

The relative positions of the glasses (horizontally long on the upperside) and the mask (lower center) in the face are mostly recognized inadvance. And the representative values of the personal differenceeigenspace coefficient vector group of the patches of the areas and thepersonal difference eigenspace coefficient vector group of the patchesof the entire face (or face areas excluding the concealment candidateareas) are compared, and it is detected that the probability that thereis no concealment is high if the representative values are similar (ifthe distance is close). Conversely, it is detected that the probabilitythat there is a concealing matter is high if the distance between therepresentative values is far.

At the positional boundary of the patch of the area, a weight asillustrated in FIG. 9 or a weight expressed by a function, such as α/x,α/x², and exp(−αx) (wherein, x denotes a distance from the concealmentcandidate position), can be added to obtain the representative value.

The uncertainty of the size of the concealing matter is taken intoaccount in the representative value provided with the weight accordingto the patch position. For example, there are various sizes in theglasses, and the glasses may or may not appear in the adjacent patchdepending on the size of the glasses. In terms of the probability, theinfluence of the glasses is greater in an area closer to the centerposition of the eyes, and the influence of the glasses is smaller if thedistance is farther (toward the periphery). Therefore, the degree ofinfluence of the concealing matter is defined as a function of thedistance from the center position of the eyes. Examples of the means forobtaining the weight include a mode of computing the weight from apredetermined function and a mode of using a look-up table (LUT) storedin advance.

If an area with a high probability of the existence of a concealingmatter is detected, a restoration based on the system of the presentinvention (restoration using the tensor projection) targeting theconcealing matter (glasses, mask, and the like) is performed for theconcealing matter area.

Example A-2-2

Although the concealing matter is detected by focusing on the distancefrom the representative value in the “example A-2-1”, the concealingmatter can also be detected from the spread of the distribution of thecoefficient vector group. More specifically, another example of theexample A-2-1 includes a mode of detecting that the probability of theexistence of the concealment is high if the distribution of the personaldifference eigenspace coefficient vector group of the patchcorresponding to the area pertinent to the concealment candidate isspread. The probability of the existence of the concealment may bedetermined to be high if the distribution of the concealment candidatearea is more spread than the distribution in the entire face.

Example A-2-3

Another example includes a mode of preliminarily obtaining a correct(image not included in the studying set) distribution shape of thepersonal difference eigenspace coefficient vector group. In this case,it is detected that the probability that there is no concealment is highif the personal difference eigenspace coefficient vector group issimilar to the preliminary distribution shape.

Example of Detecting Concealed Area for Restoration by System Differentfrom the Present Invention Example A-3-1

A mode of performing the same detection as in the “example A-2-1” forthe restoration in the concealing matter area based on anotherconversion method, such as bicubic and the “general-purposesuper-resolution processing unit 164” (see FIG. 6), is also possible.

Example of Restoration by Predicting Coefficient Vector of Area Otherthan Specific Area from Specific Area in Face Example A-4-1

The high correlation in the personal difference eigenspace may be usedfor the pixel vectors of the patch group formed by dividing the faceimage of the same person to obtain a correlation coefficient vectorgroup of the entire face from the personal difference eigenspacecoefficient vector group of only the patches of part of the face (forexample, areas of eyes, nose, and mouth).

Example A-4-1-1

For example, a representative value, such as an average value, a median,a maximum value, and a minimum value, of the personal differenceeigenspace coefficient vector group of part of the face is used as avalue of the correction coefficient vector group of the entire face.

Example A-4-1-2

In place of the “example A-4-1-1”, the distribution of the personaldifference eigenspace coefficient vector group is obtained for aplurality of patches at the center part of the face. Extrapolationprediction is performed based on the distribution to obtain thecorrection coefficient vector group of the parts other than the centerpart. For example, the distribution of the coefficient vector group isobtained for 9 (3×3) patches at the center part of the face, and thecoefficient vectors at outer positions of the nine patches are obtainedfrom the distribution based on an extrapolation method (extrapolation).

Example A-4-1-3

The distribution of the personal difference eigenspace coefficientvector group is obtained only for the patches thinned out in thehorizontal and vertical directions of the face. The distribution isinterpolated to obtain the correction coefficient vector group of thepatches for which the personal difference eigenspace coefficient vectorsare not obtained. For example, the distribution of the coefficientvector group is obtained only for the patch positions of even numbers,and the interpolation is performed to obtain the remaining patches ofodd numbers.

According to the “example A-4-1” to “example A-4-1-3”, the number ofprocesses of the [L pixel→personal difference] eigenspace projectionunit 134 is reduced from the first sub-core tensor generation unit 122described in FIG. 6, and the processes can be speeded up.

Example A-Common-1

Low-pass filtering (for example, average filtering) may be furtherapplied to the correction coefficient vector group of the patches to beprocessed and the surrounding patches. According to the mode, there isan advantageous effect of spatially smoothing the obtained correctioncoefficient vector group and removing the noise components. Maximumvalue, minimum value, or median filtering may be applied in place of theaverage filtering.

(Second Sub-Core Tensor Generation Unit)

The second sub-core tensor generation unit 124 provides the patchpositions of the output of the set value acquisition unit 120 and theconditions of the H setting to generate the sub-core tensor G_(H) fromthe LPP projection core tensor 116.

The means may be performed in the studying step in place of the mode ofperforming the means in the restoration step as in FIG. 6. Generatingthe sub-core tensor G_(H) in the studying step can reduce the processingtime of the restoration step. However, a memory for preserving thesub-core tensor G_(H) is necessary.

([Personal Difference→H Pixel] Eigenspace Projection Unit)

The [personal difference→H pixel] eigenspace projection unit 152 obtainsG_(H) from the second sub-core tensor generation unit 124 to perform theG_(H) projection described in FIG. 2( c)→(d) for the correctioncoefficient vectors of the output of the coefficient vector correctionprocessing unit 140.

(Eigenspace→H Pixel Projection Unit)

The eigenspace→H pixel projection unit 154 obtains the LPPeigenprojection matrix U_(pixels) based on the patch positions from theset value acquisition unit 120 and applies a process of the U_(pixels)projection described in (d)→(e) of FIG. 2 to the coefficient vectors ofthe output of the [personal difference→H pixel] eigenspace projectionunit 152 to obtain the high resolution image.

(Addition Unit)

The addition unit 160 outputs a sum of the input (restorationinformation of high frequency components) from the eigenspace→H pixelprojection unit 154 and the input (original low resolution enlargedimage) from the low resolution enlargement processing unit 102. Theaddition unit 160 adds and integrates all patches to generate one faceimage (high resolution image). The restoration information of the highfrequency components may be added after applying a predeterminedfiltering process to the original low resolution enlarged image.

Although the high resolution image can be obtained as described above,the weight may be added to reduce the influence of the high resolutionimage obtained from the “eigenspace→H pixel projection unit 154” whenthe correction process in the coefficient vector correction processingunit 140 is large.

Hereinafter, an example of a configuration for realizing the processwill be described.

Other than the super-resolution processing means (reference numerals100A and 100B of FIG. 6) using the LPP projection tensor,super-resolution processing means (described as “general-purposesuper-resolution processing unit 164” in FIG. 6) based on a differentalgorithm, the weight calculation unit 162, and the combining unit 166are included.

(General-Purpose Super-Resolution Processing Unit)

The general-purpose super-resolution processing unit 164 appliessuper-resolution enlargement to the input low resolution image toenlarge the image to the same size as the output.

Although the enlargement method is not particularly limited, forexample, a clustering system (Atkins, C. B.; Bouman, C. A.; Allebach, J.P., “Optimal image scaling using pixel classification”, IEEE, ImageProcessing, 2001. Proceedings. 2001 International Conference on Volume3, Issue, 2001 Page(s): 864-867 vol. 3) is used.

The clustering system is characterized by adopting a mixed model.Therefore, the super-resolution of a variety of patterns can be handledby combining a plurality of models.

The following mixed Gaussian model is assumed as means for processing.x=Σ(Ai·z+Bi)·wi(y−μi,πi)  [Expression 5]

Wherein, z: low resolution image, x: high resolution image, Ai, Bi, μi,and πi are specified during studying, and a probability wi as a weightis dynamically obtained in the restoration based on a dimension vector yof a difference between an unknown pixel and the surrounding.

Ai, Bi, μi, and πi are obtained, for example, as follows.

The center of gravity of each of 100 classes is obtained by K-means toclassify the dimension vector of difference (cluster vector) to createan initial distribution state.

Updates are repeated by an EM algorithm. The likelihood function ismaximized by the current conditional probability, and the nextconditional probability is obtained. The conditional probability isestimated in an E step. The estimation value of the E step is used tomaximize the likelihood function in an M step. The loop computation ofthe E step and the M step is continued until the output of thelikelihood function is stabilized. For example, the studying isperformed 10,000 times to study 100,000 pixels in 100 classes(convergence condition is e⁻¹⁰).

The enlargement method described in the low resolution enlargementprocessing unit 102 may be used as another enlargement method in thegeneral-purpose super-resolution processing unit 164.

(Weight Calculation Unit)

The weight calculation unit 162 is means for obtaining a weight w1 usedin the combining unit 166 to make an adjustment of increasing anddecreasing the rate of adopting the general-purpose super-resolutionsystem by the general-purpose super-resolution processing unit 164according to a deviation level of the input condition. The weight w1 isdetermined so that the rate of adopting the general-purposesuper-resolution system is reduced if the deviation level of the inputcondition is low, and the rate of adopting the general-purposesuper-resolution system is increased if the deviation level of the inputcondition is higher.

Hereinafter, a specific example of calculation in the weight calculationunit 162 will be described. It is indicated here that based on a formula([Expression 7]) of the combining unit 166 described later, the smallerthe value of the weight w1, the higher the rate of adopting (1−w1) thegeneral-purpose super-resolution system.

Example B-1-1

The means of tensor projection super-resolution (reference numerals 100Aand 100B of FIG. 6) described above have a feature that the farther thepersonal difference eigenspace coefficient vector on the personaldifference eigenspace from the coefficient vector of the studying set,the poorer the restoration characteristics (feature [1]).

FIG. 10 is a conceptual diagram illustrating the feature [1]. FIG. 10indicates the eigenspace of the tensor in a three-dimensional space andindicates the studying image vectors by dots SL₁, SL₂ . . . SL_(i). Aperiphery of the distribution range of the studying image group isindicated by reference numeral 170, and a center of gravity P_(G) of thestudying image vectors is indicated by a black circle.

Unknown image vectors IM₁, IM₂ . . . other than the studying imagevectors are indicated by white circles.

Distances are determined from the closeness of the unknown image vectorsto the studying image vector group, distances from the studying imagevectors (nearest neighbor, center of gravity, surrounding boundarypoint), inside/outside determination of the sample group (class), andthe like.

In FIG. 10, the unknown image vector indicated by IM₁ is inside thestudying set (sample group), and it is determined that the distancebetween the studying image sample and the input image is relativelyclose after comprehensively evaluating a distance dNN from a closetneighbor point (nearest neighbor), a distance dG from the center ofgravity P_(G), and a distance dAR from the surrounding boundary point(for example, evaluation values are calculated by an evaluation functionbased on a linear combination of the distances).

It is similarly determined for IM₂ that the distance from the studyingimage sample is close. The restoration of the unknown image vectors issignificantly excellent.

IM₃ and IM₄ exist inside the class of the sample group, and the distanceis a little farther compared to IM₁ and IM₂. It can be stated that IM₃and IM₄ are in a “relatively close” level. IM₃ and IM₄ can also berelatively excellently restored.

IM₅ and IM₆ exist outside of the sample group, and the distance from thestudying set is far. The restoration characteristics when the unknownimage vectors IM₅ and IM₆ are restored are low. In this way, excellentrestoration is possible if the distance from the leaning set is closer,and there is a tendency that the restoration becomes poor if thedistance is farther.

The feature [1] is used to obtain the weight w1 as follows.

The processes up to the “[L pixel→personal difference] eigenspaceprojection unit 134” of the restoration step are executed for therepresentative studying set obtained by the studying set representativevalue formation processing unit 112, and the representative personaldifference eigenspace coefficient vector group is obtained in advance.

The closest distance between the representative personal differenceeigenspace coefficient vector group and the personal differenceeigenspace coefficient vectors obtained by the “[L pixel→personaldifference] eigenspace projection unit 134” is obtained based on thepatch positions from the set value acquisition unit 120, and w1 isobtained based on an LUT as illustrated in FIG. 11B or functions such asβ1/x, β1/x², and exp(−β1x).

Example B-1-2

If the directions of the coefficient vectors of the studying set and thepersonal difference eigenspace coefficient vectors are similar, w1 isincreased.

Example B-2-1

The means of tensor projection super-resolution (reference numerals 100Aand 100B of FIG. 4) have a feature that the restoration performance ispoor if the “distribution including the number of patches as the numberof samples” of the personal difference eigenspace coefficient vectors onthe personal difference eigenspace is spread (varies) (feature [2]).

The feature [2] is used, and if the spread of the distribution of thedistance or the direction between the coefficient vectors of therepresentative studying set and the personal difference eigenspacecoefficient vectors of the patches relative to the patch samples iswide, the weight w1 is reduced. For example, a look-up table indicatingthe correspondence between the spread of the distribution and the weightw1 may be created in advance, or a function defining the correspondencemay be used for the calculation.

According to the mode, when the feature [1] of the tensor projection isused by evaluating the reliability of the method according to thepresent invention on the personal difference eigenspace of the tensor(person eigenspace of FIG. 2( c)) as compared to the pixel eigenspace ofthe tensor (image eigenspace of FIG. 2( b)), all patches can beevaluated by the same index (all patches substantially gather into onepoint), and a new advantageous effect is generated that the spread ofthe distribution can be evaluated as a reliability scale. Therefore, theweight calculation accuracy improves.

Example B-2-2

In the distribution relative to the patch samples of the example“B-2-1”, w1 is reduced for the patch samples with a small number ofsamples (or farther from the representative value). Therefore, theweight is changed according to the frequency on the histogram. In thiscase, there is an advantageous effect that the weight can be controlledpatch by patch.

Example B-3

In the distribution relative to the patch samples of the “exampleB-2-1”, the weight may be increased if the shapes of the distributionsare similar. For example, the weight is changed depending on whetherdistribution shapes of the distribution of Person A recognized in thestudying step and the distribution of the input image (unknown image)are similar.

Example B Common-1

The following configuration can be commonly adopted for the “exampleB-1-1”, “example B-1-2”, “example B-2-1”, “example B-2-2”, and “exampleB-3”. For example, a correct answer appropriateness determination indexof individual patches of each individual (for example, in the face ofPerson A) relative to the individual representative personal differencevectors as the studying samples will be further considered in the“example B-1-1” or “example B-1-2”. The distances of the individualpatches from the representative value of the distribution relative tothe patch samples are used as the determination index. The patch ishandled not suitable for a correct answer if the distance is far fromthe representative value. Specifically, wp with similar characteristicsas FIG. 11, β2/x, β2/x², exp(−β2x), and the like may be obtained, andw1′=w1·wp may be provided to the combining unit 166.

According to the mode, when the feature [1] of the tensor projection isused by evaluating the reliability of the method according to thepresent invention on the personal difference eigenspace of the tensor(personal eigenspace of FIG. 2( c)) as compared to the pixel eigenspaceof the tensor (image eigenspace of FIG. 2( b)), all patches can beevaluated by the same index (all patches substantially gather into onepoint), and a new advantageous effect is generated that the reliabilityof the studying sample defined in a tentative correct answer can beincluded in the evaluation. Therefore, the weight calculation accuracyimproves.

Example B Common-2

An average, a median, maximum, minimum, and the like may be commonlyused as the representative value in the “example B-1-1”, “exampleB-1-2”, “example B-2-1”, “example B-2-2”, and “example B-3”.

Example B Common-3

The dispersion, the standard deviation, and the like may be commonlyused as the spread (variation) of the distribution in the “exampleB-1-1”, “example B-1-2”, “example B-2-1”, “example B-2-2”, and “exampleB-3”.

Example B Common-4

If the distance between the representative value, such as the center ofgravity of the studying set and the surrounding boundary point, and thepersonal difference eigenspace coefficient vector is close or thedirections are similar, w1 is increased. According to the mode, thecalculation targets of the distance and the direction can be reduced,and the process can be speeded up.

Example B Common-5

The Euclidean distance, the Mahalanobis distance, the KL distance, andthe like can be used in the calculation of the “distance” in theexamples.

Example B Common-6

The vector angle, the inner product, the outer product, and the like canbe used in the calculation of the “direction” in the examples.

Example B Common-7

The relationship between the distance, the direction, the representativevalue, the distribution spread, the distribution shape, and arestoration error is defined as a correct answer/wrong answer set in the“studying step” described in FIG. 4. The restoration error denotes adifference between image restored by the projection function obtainedfrom the studying image set and the correct answer image, and forexample, the restoration error is indicated by an average square errorfrom the correct answer/wrong answer image or the PNSR (peak signal tonoise ratio).

The relationship between at least one of the elements “distance,direction, representative value, distribution spread, and distributionshape” and the “restoration error” and the relationship between the“restoration error” and the “weight w1” are defined in an LUT, afunction, and the like.

In the “restoration step”, the LUT or the function is used to obtain the“weight w1” from the similarly of at least one of the “distance,direction, representative value, distribution spread, and distributionshape” between the “studying step” and the “restoration step”.

A specific method of obtaining the “weight w1” from the similarity of atleast one of the “distance, direction, representative value,distribution spread, and distribution shape” will be illustrated below.

<Process in Studying Step>

The relationship between at least one of the “distance, direction,representative value, distribution spread, distribution shape” and the“restoration error” is obtained. For example, “characteristics ofdistance−restoration error” is obtained. Characteristics with reliableprobability proportional to the frequency may be obtained.

<Process in Restoration Step>

The closest “distance, direction, representative value, distributionspread, and distribution shape” in the “studying step” from the“distance, direction, representative value, distribution spread, anddistribution shape” obtained in the “restoration step” described in FIG.6 is selected to obtain a corresponding “restoration error”.

Based on the selected “restoration error”, the “weight” is obtained fromthe relationship of the following formula ([Expression 6]). It isassumed here that the smaller the “restoration error”, the greater the“weight”.Weight w1=b0+b1×(restoration error)

In place of the linear function indicated in [Expression 6], a nonlinearfunction may be defined to obtain the weight.

Example B Common-8

In the function defining the correlation between at least one of the“distance, direction, representative value, distribution spread, anddistribution shape” of the correct answer/wrong answer set on thepersonal difference eigenspace in the “example B-common-7” and the“weight”, the coefficients b0 and b1 of [Expression 5] may be obtainedby (regularization) least squares method, multiple regression analysis,SVM (regression), AdaBoost (regression), non-parametric Bayes, maximumlikelihood estimation method, EM algorithm, variational Bayes method,Markov chain Monte Carlo method, and the like.

Example B Common-9

In the examples (“example B-1-1” to “example B-common-8”), low-pass(average) filtering may be further applied to the patches to beprocessed and the weight of the surrounding patches. The mode has anadvantageous effect of spatially smoothing the obtained weight and anadvantageous effect of removing the noise. The maximum value, theminimum value, and the median filter may also be applied.

The “examples B-common-1 to 9” methods can also be applied to the weightin the coefficient vector correction processing unit 140 describedabove.

As described, in the configuration of using the image conversion meansof another system (means of general-purpose super-resolution here)according to the deviation level of the input image relative to thestudying image set (deviation level of input condition), if therepresentative value of the studying image set is used when using thepositional relationship between the coefficient vectors on theeigenspace, there is an advantageous effect that the utilizationfunction of the other system can be effectively functioned.

(Combining Unit)

The combining unit 166 of FIG. 6 combines or selects the image providedfrom the adding unit 160 (input image 1) and the image provided from thegeneral-purpose super-resolution processing unit 164 (input image 2)according to the following weight obtained in the weight calculationunit 162.Output high resolution image=Σ(wi·Ii)=w1·I1+w2·I2

Wherein, w1 denotes the weight w1 of an output I1 of the adding unit160, and w2 denotes a weight w2=1−w1 of an output I2 of thegeneral-purpose super-resolution processing unit 164.

According to the image processing system with the configuration, a highquality image can be obtained from a low quality input image. A robusthigh image quality formation process with a wide tolerance for the inputcondition can be realized.

In addition to the general-purpose super-resolution processing unit 164,one or a plurality of high image quality formation processing unitsbased on another method may be further arranged. The processing unitsmay be selectively used, or the images may be combined by arbitraryweighting.

Meanwhile, the reliability of the super-resolution restoration processmay be extremely low depending on the condition of the input image, andthere may be a case in which it is desirable to output the image usingthe information of the original input image instead of outputting thefailed image with low reliability. Therefore, a processing unit thatsimply enlarges the input image may be arranged in place of or inaddition to the general-purpose super-resolution processing unit 164,and an image enlarged by the enlargement processing unit (image notsubjected to the super-resolution restoration process) may be providedto the combining unit 166.

Modified Example 1 of Embodiment

FIG. 12 is a block illustrating another embodiment. In FIG. 12, the sameor similar elements as in the configuration of FIG. 7 are designatedwith the same reference numerals, and the description will not berepeated.

The embodiment illustrated in FIG. 12 is a mode of generating a firstsub-core tensor 123 and a second sub-core tensor 125 and storing andpreserving the tensors in storage means such as a memory in the studyingstep.

The LPP eigenprojection matrix U and the projection core tensor G (aswell as the first sub-core tensor 123 and the second sub-core tensor 125generated from the matrix and the tensor) can be repeatedly used in thesubsequent processes once the matrix and the tensor are created andpreserved. Therefore, a mode of forming parameters from the matrix andthe tensor for each studying image set and arbitrarily resettingappropriate projection matrix and tensor according to the content of theinput image in the restoration step is preferable.

For example, parameters are formed by country or region from projectionconversion sets, such as a set of the projection matrix and the tensorgenerated based on the studying image set of the faces of Japanese and aset of the projection matrix and the tensor generated based on thestudying image set of the faces of Westerners, and the parameters areswitched and used as necessary.

Alternatively, the set of the projection matrix and the tensor may beswitched based not only on the process of the super-resolutionrestoration of the face image, but also on applications of the process.For example, the studying image set is switched according to anapplication, such as for endoscopic image and for vehicle image, togenerate the LPP eigenprojection matrix U and the projection core tensorG (as well as the first sub-core tensor 123 and the second sub-coretensor 125 generated from the matrix and the tensor), and the generatedprojection matrix and the tensor are preserved and stored in anon-volatile memory, a magnetic disk, or other storage means. Variousimage processing is possible with the same algorithm by reading andsetting the projection matrix and the tensor according to theapplication.

Modified Example 2 of Embodiment

Although the configurations that can carry out the studying step and therestoration step by one image processing device are illustrated in FIGS.6 and 12, the image processing device that carries out the studying stepand the image processing device that carries out the restoration stepcan be separately arranged. In this case, it is desirable that the imageprocessing device that manages the restoration step can acquireinformation of the separately created projection relationship(eigenprojection matrix and projection tensor) from the outside. A mediainterface or a communication interface corresponding to an optical diskor other removable storage media can be applied as the informationacquisition means.

Modified Example 3 of Embodiment

Although the LPP is illustrated as the projection that uses the localrelationship in the embodiment, in place of the LPP, various manifoldstudying methods can also be applied, such as locally linear embedding(LLE), linear tangent-space alignment (LTSA), Isomap, Laplacian eigenmap(LE), and neighborhood preserving embedding (NPE).

The technique for obtaining the representative studying image group ofthe present invention is applied not only to the projection using thelocal relationship, but also to the tensor singular value decomposition(TSVD) and the like.

Modified Example 4 of Embodiment

In the embodiment described in FIG. 6, the modalities of the patch andthe resolution of the four types of modalities described in Table 1 arehandled as known elements to set the conditions to simplify thedescription, and the projection route from the pixel real space throughthe pixel eigenspace and the personal difference eigenspace is designedby focusing on the modalities of the “pixel value” and the “personaldifference”. However, the design of the projection route is not limitedto the present example in carrying out the present invention. Variouseigenspaces can be selected as the eigenspaces in the projection routeaccording to the modality variations.

Modified Example 5 of Embodiment

The image of the conversion source input to the restoration step may bean image area partially cut out (extracted) from an image at a prestagebefore the procedure of the processes described in FIGS. 6 and 12. Forexample, a process of extracting a face part of a person from anoriginal image is executed, and the extracted face image area can behandled as the input image data of the restoration step.

Processing means for executing a combining process of replacing theextracted area by the restored output high resolution image andembedding the image to the original image may be added. In this case,the enlargement magnification is adjusted in accordance with the size ofthe ultimate output image (or the size of the background to becombined).

<Other Applications>

Applications to various “targets”, “modalities”, and “image processing”are possible by changing the studying image set as follows. Therefore,the application range of the present invention is not limited to theembodiments described above.

Other than the face, the image as the “target” may be an area includinga region of part of a human body, such as head or hands of a person, ora region including at least part of a living body other than the humanbody. The living body includes specific tissues inside the living body,such as blood vessels in the living body. When the image processingtechnique of the present invention is applied to an endoscopic system,tumor tissues in the living body may be included in the concept of the“living body”, and the tumor tissues may be the “target”.

Other than the living body, a card, such as currency and a cash card, avehicle, and a number plate of a vehicle as well as a text, a drawing, atable, and a photograph of a document scanned by a scanner device suchas a copy machine can also be the targets.

The “modality” includes the direction of the subject, the size, theposition, the illumination position, and the like. Other examples of the“modality” include the type of the subject, such as race, age, and sex,and the attribute of the subject image, such as the expression of theimaged person, the gesture of the imaged person, the posture of theimaged person, and the wearing material worn by the imaged person.Examples of the wearing material include glasses, sunglasses, a mask,and a hat.

Other than the super-resolution formation, the “image processing” thatcan apply the present invention includes projection processes, such as aprocess of reducing aliasing components, multi-color formation,multi-gradation formation, noise reduction, artifact reduction forreducing artifacts such as block noise and mosquito noise, blurreduction, sharpness formation, high frame rate formation, wide dynamicrange formation, color gradation correction, distortion aberrationcorrection, and encoding. For example, in the case of the noisereduction, a noise image (equivalent to the “low quality image”) and animage without noise (equivalent to the “high quality image”) are handledas a pair to study the projection relationship.

The present invention can be applied not only to still images, but alsoto frame images (or field images) that constitute a video.

<Application to Monitoring System>

FIG. 13 illustrates an example of an image processing system 200according to an embodiment of the present invention. The imageprocessing system 200 described below can function as, for example, amonitoring system.

The image processing system 200 includes a plurality of imaging devices210 a-d that image a monitoring target space 202, an image processingdevice 220 that processes the photographic images taken by the imagingdevices 210 a-d, a communication network 240, an image processing device250, an image database (DB) 255, and a plurality of display devices 260a-e. The image processing device 250 can be installed in a space 205different from the monitoring target space 202 (for example, a locationfar away from the monitoring target space 202), and the display devices260 a-e can be installed in a space 206 different from the monitoringtarget space 202 and the installation space 205 of the image processingdevice 250.

The imaging device 210 a includes an imaging unit 212 a and aphotographic image compression unit 214 a. The imaging unit 212 acontinuously images the monitoring target space 202 to take a pluralityof photographic images. The photographic images obtained by the imagingunit 212 a may be photographic images in an RAW format. The photographicimage compression unit 214 a synchronizes the photographic images in theRAW format taken by the imaging unit 212 a and generates video data bycompressing a video including the plurality of photographic imagesobtained by the synchronization based on MPEG encoding or other encodingsystems. The imaging device 210 a outputs the generated video data tothe image processing device 220.

The other imaging devices 210 b, 210 c, and 210 d have the sameconfigurations as the imaging device 210 a, and the video data generatedby the imaging devices 210 a-d are transmitted to the image processingdevice 220. In the following description, the imaging devices 210 a-dmay be collectively called “imaging device 210”. Similarly, the displaydevices 260 a-e may be collectively called “display device 260”. In thefollowing description, characters following numeric codes, such asalphabetical characters at the end of reference numerals provided tosimilar constituent elements, may be abbreviated to collectively callthe constituent elements indicated by the numeric codes.

The image processing device 220 decodes the video data acquired from theimaging device 210 to acquire the video. The image processing device 220detects a plurality of feature areas with different types of features,such as an area of an imaged person 270 and an area of an imaged movingbody 280 such as a vehicle, from the plurality of photographic imagesincluded in the acquired video. The image processing device 220compresses the images of the feature areas at a strength according tothe type of the features and compresses the images of the areas otherthan the feature areas at a strength stronger than the compressivestrength for compressing the images of the feature areas.

The image processing device 220 also generates feature area informationincluding information for specifying the feature areas detected from thephotographic images. The feature area information may be text dataincluding the positions of the feature areas, the sizes of the featureareas, the number of feature areas, and identification information foridentifying the photographic images from which the feature areas aredetected or may be data obtained by applying processing, such ascompression and encryption, to the text data.

The image processing device 220 attaches the generated feature areainformation to the compressed video data and transmits the informationto the image processing device 250 through the communication network240.

The image processing device 250 receives the compressed video dataassociated with the feature area information from the image processingdevice 220. The image processing device 250 stores the compressed videodata in the image DB 255 in association with the feature areainformation associated with the compressed video data. As for the imageDB 255, the compressed video data may be stored in a non-volatilestorage medium such as a hard disk. In this way, the image DB 255 storesthe compressed photographic images.

In response to a request from the display device 260, the imageprocessing device 250 reads out the compressed video data and thefeature area information from the image DB 255, uses the accompanyingfeature area information to expand the read out compressed video data togenerate a video for display, and transmits the video to the displaydevice 260 through the communication network 240. The display device 260includes a user interface that can input a search condition of the imageand the like. The display device 260 can transmit various requests tothe image processing device 250 and displays the video for displayreceived from the image processing device 250.

In place of or in combination with the video display, the imageprocessing device 250 can specify photographic images satisfying varioussearch conditions and feature areas of the photographic images based onthe positions of the feature areas included in the feature areainformation, the sizes of the feature areas, and the number of featureareas. The image processing device 250 may decode the specifiedphotographic images and provide the images to the display device 260 todisplay the images that match the requested search conditions on thedisplay device 260.

The image processing device 250 may use the corresponding feature areainformation to expand the compressed video data acquired from the imageprocessing device 220 to generate a video for display and then store thevideo in the image DB 255. At this time, the image processing device 250may store the video for display in the image DB 255 in association withthe feature area information. According to the mode, the imageprocessing device 250 can read out the video for display (expanded) fromthe image DB 255 according to the request from the display device 260 totransmit the video to the display device 260 along with the feature areainformation.

In place of the mode of providing the expanded video for display fromthe image processing device 250 to the display device 260, the displaydevice 260 may execute the expansion process of the compressed videodata to generate the image for display. More specifically, the displaydevice 260 may receive the feature area information and the compressedvideo data from the image processing device 250 or the image processingdevice 220. In the mode, when the received compressed video data isdecoded and displayed on the display device 260, the display device 260may temporarily and simply enlarge the feature areas in the decodedphotographic images and display the feature areas on the display device260.

The display device 260 may further determine the image quality of thefeature areas according to the processing capacity in the display device260 to increase the image quality of the images of the feature areasbased on the determined image quality. The display device 260 mayreplace the images of the feature areas in the photographic imagesdisplayed on the display device 260 by the images of the feature areasformed by increasing the image quality and display the images on thedisplay device 260. The super-resolution means using the tensorprojection of the present invention can be used as the processing meansof high image quality formation upon the replacement and display. Morespecifically, the image processing device applying the present inventioncan be loaded on the display device 260.

According to the image processing system 200 of the present example, theinformation indicating the feature areas is stored in association withthe video. Therefore, a photographic image group that matches apredetermined condition in the video can be quickly searched and cued.According to the image processing system 200 of the present example,only the photographic image group that matches the predeterminedcondition can be decoded. Therefore, the replay instruction can bereadily responded, and a partial video that matches the predeterminedcondition can be quickly displayed.

A recording medium 290 illustrated in FIG. 13 stores programs for theimage processing device 220, the image processing device 250, and thedisplay device 260. The programs stored in the recording medium 290 areprovided to electronic information processing devices, such ascomputers, that respectively function as the image processing device220, the image processing device 250, and the display device 260 of thepresent embodiment. CPUs included in the computers operate according tothe contents of the programs and control the components of thecomputers. The programs executed by the CPUs cause the computers tofunction as the image processing device 220, the image processing device250, the display device 260, and the like described in relation to FIG.13 and the following drawings.

Examples of the recording medium 290 include a CD-ROM, as well as anoptical recording medium, such as a DVD or a PD, a magneto-opticalrecording medium, such as an MO or an MD, a magnetic recording medium,such as a tape medium or a hard disk device, a semiconductor memory, anda magnetic memory. A storage device, such as a hard disk or a RAM,arranged on a server system connected to a dedicated communicationnetwork or to the Internet can also function as the recording medium290.

Hereinafter, examples of configurations of the image processing devices220 and 250 and the display device 260 in the image processing system200 of the present example will be described in further detail.

[Description of Image Processing Device 220]

FIG. 14 illustrates an example of a block configuration of the imageprocessing device 220. The image processing device 220 includes an imageacquisition unit 222, a feature area specifying unit 226, an externalinformation acquisition unit 228, a compression control unit 230, acompression unit 232, an association processing unit 234, and an outputunit 236. The image acquisition unit 222 includes a compressed videoacquisition unit 223 and a compressed video expansion unit 224.

The compressed video acquisition unit 223 acquires the encoded videodata generated by the imaging device 210 (see FIG. 13). The compressedvideo expansion unit 224 expands the video data acquired by thecompressed video acquisition unit 223 to generate a plurality ofphotographic images included in the video. Specifically, the compressedvideo expansion unit 224 decodes the encoded video data acquired by thecompressed video acquisition unit 223 to extract the plurality ofphotographic images included in the video. The photographic imagesincluded in the video may be frame images or field images.

The plurality of photographic images obtained by the compressed videoexpansion unit 224 are supplied to the feature area specifying unit 226and the compression unit 232. The feature area specifying unit 226detects a feature area from the video including the plurality ofphotographic images. Specifically, the feature area specifying unit 226detects the feature area from each of the plurality of photographicimages.

For example, the feature area specifying unit 226 detects an image area,in which the image content changes in the video, as the feature area.Specifically, the feature area specifying unit 226 may detect an imagearea including a moving object as the feature area. The feature areaspecifying unit 226 can detect a plurality of feature areas withdifferent types of features from each of the plurality of photographicimages.

The types of the features may be types classified based on the types ofthe objects, such as a person and a moving body. The types of theobjects may be determined based on the shapes of the objects or thedegree of coincidence of the colors of the objects. In this way, thefeature area specifying unit 226 may detect the plurality of featureareas including different types of objects from the plurality ofphotographic images.

Example 1 of Feature Area Detection Method

For example, the feature area specifying unit 226 may extract an objectthat matches with a predetermined shape pattern at a degree ofcoincidence greater than a predetermined degree of coincidence from theplurality of photographic images and detect areas in the photographicimages including the extracted object as the feature areas with the sametype of feature. A plurality of shape patterns may be defined for eachtype of the features. An example of the shape pattern includes a shapepattern of the face of a person. Patterns of different faces may bedefined for each of a plurality of persons. In this way, the featurearea specifying unit 226 can detect different areas including differentpersons as different feature areas.

Other than the face of a person, the feature area specifying unit 226can detect, as the feature areas, areas including a region of part of ahuman body, such as the head of a person or hands of a person, or aregion of at least part of a living body other than the human body.

When images inside a living body are processed, such as in anapplication to an endoscopic system with a similar configuration as theimage processing system 200, specific tissues inside the living body,such as blood vessels inside the living body, or tumor tissues insidethe living body can be the target. Other than the living body, thefeature area specifying unit 226 may detect, as the feature areas, areasof an imaged card, such as currency and cash card, an imaged vehicle,and an imaged number plate of a vehicle.

Example 2 of Feature Area Detection Method

Other than the pattern matching based on template matching and the like,the feature area specifying unit 226 can detect the feature areas basedon, for example, a studying result based on machine studying (forexample, AdaBoost) described in Japanese Patent Application Laid-OpenNo. 2007-188419. For example, an image feature amount extracted from animage of a predetermined subject and an image feature amount extractedfrom an image of a subject other than the predetermined subject are usedto study the features of the image feature amount extracted from theimage of the predetermined subject. The feature area specifying unit 226may detect areas, from which the image feature amount including featuresthat match the studied features is extracted, as the feature areas.

Other than the examples 1 and 2, the feature areas can be detected byvarious methods, and the feature area specifying unit 226 detects aplurality of feature areas from a plurality of photographic imagesincluded in each of a plurality of videos based on an arbitrary method.The feature area specifying unit 226 supplies information indicating thedetected feature areas to the compression control unit 230. Theinformation indicating the feature areas can include coordinateinformation of the feature areas indicating the positions of the featureareas, type information indicating the types of the feature areas, andinformation for identifying a video in which the feature areas aredetected.

The compression control unit 230 controls the compression process ofvideo by the compression unit 232 based on information indicating thefeature areas acquired from the feature area specifying unit 226. Basedon the control by the compression control unit 230, the compression unit232 compresses the photographic images at different strengths in thefeature areas of the photographic images and in the areas other than thefeature areas of the photographic images. For example, the compressionunit 232 reduces the resolution of the areas other than the featureareas in the photographic images included in the video compared to theresolution in the feature areas to compress the photographic images. Inthis way, the compression unit 232 compresses the image areas in thephotographic images at strengths according to the importance of theimage areas.

When the feature area specifying unit 226 detects a plurality of featureareas, the compression unit 232 may compress the images of the pluralityof feature areas in the photographic images at strengths according tothe types of the features of the feature areas. For example, thecompression unit 232 may reduce the resolutions of the images of theplurality of feature areas in the photographic images to resolutionsdefined in accordance with the types of the features of the featureareas.

The association processing unit 234 associates the information forspecifying the feature areas detected from the photographic images withthe photographic images. Specifically, the association processing unit234 associates the information for specifying the feature areas detectedfrom the photographic images with the compressed video including thephotographic images as video configuration images. The output unit 236outputs, to the image processing device 250, the compressed video dataincluding the information for specifying the feature areas associated bythe association processing unit 234.

The external information acquisition unit 228 acquires, from the outsideof the image processing device 220, data used in the process by thefeature area specifying unit 226 to specify the feature areas. Thefeature area specifying unit 226 uses the data acquired by the externalinformation acquisition unit 228 to specify the feature areas. The dataacquired by the external information acquisition unit 228 will bedescribed in relation to a parameter storage unit 650 illustrated inFIG. 15 described below.

Example of Configuration of Feature Area Specifying Unit 226

FIG. 15 illustrates an example of a block configuration of the featurearea specifying unit 226. The feature area specifying unit 226 includesa first feature area specifying unit 610, a second feature areaspecifying unit 620, an area estimation unit 630, a high image qualityformation area determination unit 640, the parameter storage unit 650,and an image generation unit 660. The second feature area specifyingunit 620 includes a partial area determination unit 622 and a featurearea determination unit 624.

The first feature area specifying unit 610 acquires the photographicimages as the video configuration images included in the video from theimage acquisition unit 222 and specifies the feature areas from theacquired photographic images. The first feature area specifying unit 610may use the detection method illustrated in the “example 1 and example 2of feature area detection method” to detect the feature areas to specifythe feature areas from the photographic images.

The image generation unit 660 generates, from the photographic images,high quality images, in which the image quality is further increased inthe areas with a higher possibility of being specified as the featureareas among the areas not specified as the feature areas (equivalent to“first feature areas”) by the first feature area specifying unit 610.The super-resolution image processing means using the tensor projectionaccording to the present invention can be used as the means forgenerating the high quality images by the image generation unit 660.

The second feature area specifying unit 620 searches the feature areas(equivalent to “second feature areas”) from the high quality imagesgenerated by the image generation unit 660. The feature areas specifiedby the first feature area specifying unit 610 and the second featurearea specifying unit 620 are supplied to the compression control unit230 as the feature areas specified by the feature area specifying unit226.

The second feature area specifying unit 620 may search the feature areasin further detail compared to the first feature area specifying unit 610based on the high quality images obtained from the image generation unit660. For example, the second feature area specifying unit 620 mayinclude a detector that detects the feature areas more accurately thanthe detection accuracy for specifying the feature areas by the firstfeature area specifying unit 610. More specifically, the detectorcapable of more accurate detection than the detection accuracy of thedetector included as the first feature area specifying unit 610 may beprovided as the second feature area specifying unit 620.

In another mode, the second feature area specifying unit 620 may searchthe feature areas in further detail compared to the first feature areaspecifying unit 610 from the same input images (images not subjected tothe high image quality formation process) as the images input to thefirst feature area specifying unit 610.

The image generation unit 660 may generate, from the photographicimages, high quality images, in which the image quality ispreferentially increased in the areas with a higher possibility of beingspecified as the feature areas among the areas not specified as thefeature areas by the first feature area specifying unit 610. The imagegeneration unit 660 may also generate the high quality images based onimage processing to the photographic images.

The image generation unit 660 may generate, from the photographicimages, the high quality images, in which the image quality is furtherincreased in the areas with a higher possibility of being specified asthe feature areas among the areas not specified as the feature areas bythe first feature area specifying unit 610, after the first feature areaspecifying unit 610 has specified the feature areas. In this way, the“areas not specified as the feature areas by the first feature areaspecifying unit 610” may be the areas not specified as the feature areasby the first feature area specifying unit 610 at the stage when thefirst feature area specifying unit 610 has specified the feature areas.In this case, the second feature area specifying unit 620 will searchthe feature areas again.

In addition, the “areas not specified as the feature areas by the firstfeature area specifying unit 610” may be areas predicted to be notspecified by the first feature area specifying unit 610 at the stagewhen the first feature area specifying unit 610 has not specified thefeature areas. For example, when the first feature area specifying unit610 detects the areas that match a predetermined condition as thefeature areas, the “areas not specified as the feature areas by thefirst feature area specifying unit 610” may be areas that do not meetthe condition. The image generation unit 660 may generate the highquality images at the stage when the first feature area specifying unit610 has not specified the feature areas.

Although the first feature area specifying unit 610 and the secondfeature area specifying unit 620 are illustrated by different functionalblocks in the present block diagram (FIG. 15), it is obvious that theunits can be included as a single functional element. For example, thefirst feature area specifying unit 610 and the second feature areaspecifying unit 620 can at least partially share a hardware element,such as an electric circuit for feature area detection, and a softwareelement, such as software for feature area detection.

Although the image generation unit 660 generates images formed byincreasing the image quality of the input images in the exampledescribed above, the image generation unit 660 may generate images withhigher image quality than the images subjected to the feature areaspecifying process for specifying the feature areas executed by thefirst feature area specifying unit 610 and provide the images to thesecond feature area specifying unit 620. For example, when the firstfeature area specifying unit 610 applies predetermined image processingto the input images to specify the feature areas, the image generationunit 660 may generate the images with higher image quality than theimages obtained by the image processing and provide the images to thesecond feature area specifying unit 620.

It is sufficient if the high quality images generated by the imagegeneration unit 660 are images with higher image quality than the imagesused by the first feature area specifying unit 610 in the feature areaspecifying process, and the high quality images include images withhigher image quality than the input images and images with lower imagequality than the input images. In this way, the image generation unit660 generates, from the input images, high quality images obtained bychanging the image quality of the areas not specified as the featureareas by the first feature area specifying unit 610 to image qualitycorresponding to the possibility of being specified as the featureareas. The image generation unit 660 may also generate high qualityimages with the image quality of accuracy corresponding to thepossibility of being specified as the feature areas.

The area estimation unit 630 estimates areas that should be specified asthe feature areas in the photographic images. For example, if thefeature area specifying unit 226 should specify areas of a moving objectin the video as the feature areas, the area estimation unit 630estimates the areas with the moving object in the video. For example,the area estimation unit 630 estimates the position of the moving objectbased on the position of the moving object extracted from other one ormore photographic images as video configuration images included in thesame video, the timing that the other photographic images are taken, andthe like. The area estimation unit 630 may estimate areas in apredetermined size including the estimated position as the areas withthe moving object in the video.

In this case, the first feature area specifying unit 610 specifies, asthe feature areas, the areas of the moving object among the areasestimated by the area estimation unit 630 in the photographic images.The image generation unit 660 may generate high quality images obtainedby increasing the image quality of the areas, in which the areas of themoving object are not specified by the first feature area specifyingunit 610, among the areas estimated by the area estimation unit 630.

As a result, the possibility that the moving object can be extracted byre-searching increases when the moving object cannot be detected fromthe areas with a high possibility of the existence of the moving object.In this way, the probability of a missed detection of the feature areasby the feature area specifying unit 226 can be reduced.

The partial area determination unit 622 determines whether the images ofone or more partial areas at predetermined positions in specific imageareas meet predetermined conditions. The feature area determination unit624 determines whether the specific image areas are the feature areasbased on the determination results of the partial area determinationunit 622. For example, in the determination of whether the specificimage areas are the feature areas, the partial area determination unit622 determines whether the plurality of different partial areas on thespecific image areas meet the predetermined conditions. The feature areadetermination unit 624 determines that the specific image areas are thefeature areas when the number of partial areas from which thedetermination result of “no” is obtained is smaller than a predeterminedvalue.

When the second feature area specifying unit 620 determines whether thespecific image areas are the feature areas based on the process for theone or more partial areas at the predetermined positions in the specificimage areas, the image generation unit 660 may increase the imagequality of the one or more partial areas when generating the highquality images obtained by increasing the image quality of the specificimage areas. This can increase the image quality of only the areaseffective for the feature area detection process, and the amount ofcomputation in the re-detection process of the feature areas can bereduced.

The high image quality formation area determination unit 640 determinesareas for which the image generation unit 660 increases the imagequality. Specifically, the high image quality formation areadetermination unit 640 determines wider areas as the areas for which theimage generation unit 660 increases the image quality when thepossibility that the areas are specified as the feature areas is lower.The image generation unit 660 generates high quality images formed byfurther increasing the image quality of the areas determined by the highimage quality formation area determination unit 640. As a result, thepossibility that the moving object can be extracted by re-searching canbe increased, and the probability of a missed detection of the featureareas by the feature area specifying unit 226 can be reduced.

The parameter storage unit 650 stores image processing parameters usedto increase the image quality of the images in association with theamount of features extracted from the images. The image generation unit660 uses the image processing parameters stored in the parameter storageunit 650 in association with the amount of features that matches theamount of features extracted from the target areas of the high imagequality formation to generate the high quality images in which the imagequality of the target areas of the high image quality formation isincreased. The parameter storage unit 650 may store the image processingparameters calculated by studying using a plurality of images, fromwhich similar amounts of features are extracted, as teacher images inassociation with the amount of features that represents the similaramounts of features.

The image processing parameters may be image data including spatialfrequency components of a higher frequency area that should be added tothe image data as a target of the high image quality formation. Whendata of pixel values of a plurality of pixels or data of a plurality offeature amount components is handled as input data, other examples ofthe image processing parameters include a vector, a matrix, a tensor, ann-dimensional mixed normal distribution, an n-dimensional mixedmultinomial distribution, and the like for converting the input data todata indicating the high quality images. It is assumed that n is aninteger one or greater. The image processing parameters will bedescribed later in relation to the operation of the image processingdevice 250.

The external information acquisition unit 228 illustrated in FIG. 13acquires at least one of the image processing parameters and the amountof features stored in the parameter storage unit 650 (described in FIG.15) from the outside. The parameter storage unit 650 stores at least oneof the image processing parameters and the amount of features acquiredby the external information acquisition unit 228.

FIG. 16 illustrates an example of a specifying process of feature areasin the feature area specifying unit 226. A process of specifying featureareas in a photographic image 700 will be described here.

As illustrated in FIG. 16, the first feature area specifying unit 610(see FIG. 15) calculates, for a plurality of image areas of thephotographic image 700, matching degrees to a predetermined condition.The first feature area specifying unit 610 specifies an area 710-1 andan area 710-2, in which the matching degrees to the predeterminedcondition in the photographic image is greater than a first threshold,as the feature areas.

The high image quality formation area determination unit 640 (see FIG.15) selects an area 710-3 and an area 710-4 in which the matchingdegrees to the predetermined condition in the photographic image isgreater than a second threshold which is smaller than the firstthreshold (see FIG. 16). The high image quality formation areadetermination unit 640 determines an area 710-5, which includes the area710-3 and has a size corresponding to the matching degree of the imageof the area 710-3 for the condition, as a target area of the high imagequality formation by the image generation unit 660. The high imagequality formation area determination unit 640 also determines an area710-6, which includes the area 710-4 and has a size corresponding to thematching degree of the image of the area 710-4 for the condition, as atarget area of the high image quality formation by the image generationunit 660.

It is determined that a smaller matching degree is calculated for thearea 710-4 compared to the area 710-3 in the example of FIG. 16, and thehigh image quality formation area determination unit 640 determines thearea 710-6, which is enlarged from the area 710-4 at a greaterenlargement rate, as the target area of the high image quality formationby the image generation unit 660 (see FIG. 15). In this way, the highimage quality formation area determination unit 640 determines the area,which is obtained by enlarging an area, in which the matching degree tothe condition is greater than the predetermined second threshold, at anenlargement rate according to the matching degree, as the target area ofthe high image quality formation by the image generation unit 660.

The second feature area specifying unit 620 (see FIG. 15) searches afeature area from the images of the area 710-5 and the area 710-6 withan increased image quality (see FIG. 16). The second feature areaspecifying unit 620 may execute the same process as the first featurearea specifying unit 610 to search an area that meets the condition fromimages of the area 710-5 and the area 710-6 with an increased imagequality. It is assumed here that the second feature area specifying unit620 has determined that an area 722 meets the condition in an image 720of the area 710-5 with an increased image quality. In this case, thefeature area specifying unit 226 specifies an area 710-7 correspondingto the area 722 on the image 720 as a feature area in addition to thearea 710-1 and the area 710-2 specified by the first feature areaspecifying unit 610.

In this way, the image generation unit 660 (see FIG. 15) generates, fromthe photographic image, a high quality image formed by increasing theimage quality of an area with a greater matching degree to apredetermined condition among the areas not specified as the featureareas by the first feature area specifying unit 610. Specifically, theimage generation unit 660 generates a high quality image formed byincreasing the image quality of an area in which the matching degree tothe condition is greater than the predetermined second threshold amongthe areas not specified as the feature areas by the first feature areaspecifying unit 610. As a result, the possibility of extracting thefeature areas from areas that are likely to be the feature areas can beincreased, and the probability of a missed detection of the featureareas can be reduced.

As described, the areas except the areas specified as the feature areasby the first feature area specifying unit 610 and the target areas ofthe high image quality formation are determined as non-feature areasthat are not the feature areas. Based on results of specifying thefeature areas by the first feature area specifying unit 610 and thesecond feature area specifying unit 620, preliminary test results,posterior test results, or the like, the value of the first thresholdmay be set so that the probability of specifying an area, which is not afeature area, as a feature area is greater than a predetermined value.This can reduce the possibility that the areas specified as the featureareas by the first feature area specifying unit 610 include non-featureareas. Although a matching degree close to the first threshold may becalculated in the non-feature areas, the possibility that the areas arefalsely detected as the feature areas can be reduced by setting thefirst threshold as described above.

Based on results of specifying the feature areas by the first featurearea specifying unit 610 and the second feature area specifying unit620, preliminary test results, posterior test results, or the like, thevalue of the second threshold may be set so that the matching degreescalculated from the feature areas are equal to or greater than thesecond threshold. This can reduce the possibility that the areas, inwhich the matching degrees equal to or smaller than the second thresholdare calculated, include the feature areas. Although a matching degreeclose to the second threshold may be calculated for the feature areas,the possibility that the areas are determined as non-feature areas canbe reduced by setting the second threshold as described above.

Meanwhile, as a result of setting the first threshold and the secondthreshold, there is a possibility that areas, in which the matchingdegrees greater than the second threshold and equal to or smaller thanthe first threshold are calculated, include the feature areas. Accordingto the feature area specifying unit 226, the image quality of the areasis increased, and then the second feature area specifying unit 620searches the feature areas. Therefore, the feature areas and thenon-feature areas can be appropriately separated, and the probability ofa missed detection of the feature areas and the probability of detectinga non-feature area as a feature area can be reduced. In this way, thefeature area specifying unit 226 can provide a feature area detectorwith high sensitivity and specificity.

Other than determining whether to execute the high image qualityformation process based on the relationship between the matching degreeand the threshold, the image generation unit 660 may generate a highquality image formed by increasing the image quality of at least part ofthe image areas of the input image at high image quality formationaccuracy corresponding to the matching degree to the condition. In thiscase, the high image quality formation accuracy may be defined by acontinuous function or a discontinuous function corresponding to thematching degree.

FIG. 17 illustrates another example of the specifying process of thefeature areas in the feature area specifying unit 226. An example of aprocess by the feature area specifying unit 226 for specifying an areaof a moving object from a video as a feature area is particularlyillustrated here.

It is assumed that the first feature area specifying unit 610 or thesecond feature area specifying unit 620 (see FIG. 15) has specified anarea 810-1 and an area 810-2 as the feature areas in a photographicimage 800-1 and a photographic image 800-2, respectively, as illustratedin FIG. 17. It is assumed here that the area 810-1 and the area 810-2include objects of the same imaged subject.

In this case, the area estimation unit 630 (see FIG. 15) determines anarea 810-3 as an area where an object of the same subject should existin a photographic image 800-3 based on the positions on the images ofthe area 810-1 and the area 810-2, the timing of imaging of thephotographic image 800-1 and the photographic image 800-2, and thetiming of imaging of the photographic image 800-3 (FIG. 17). Forexample, the area estimation unit 630 calculates the speed of the movingobject on the image areas from the positions of the area 810-1 and thearea 810-2 on the images and the timing of imaging of the photographicimage 800-1 and the photographic image 800-2 and determines that thearea 810-3 is an area where an object of the same subject should existbased on the calculated speed, the position of the area 810-2, and thetime difference between the timing of imaging of the photographic image800-2 and the timing of imaging of the photographic image 800-3.

The first feature area specifying unit 610 (see FIG. 15) searches amoving object from the area 810-3 (FIG. 17). If the first feature areaspecifying unit 610 does not detect a moving object from the area 810-3,the image generation unit 660 generates a high quality image 820-4formed by increasing the image quality of the area 810-3 (FIG. 17). Thesecond feature area specifying unit 620 searches a moving object fromthe high quality image 820-4. This can increase the possibility of theextraction of the object from an area with a high possibility of thedetection of the moving object and can reduce the probability of amissed detection of the moving object.

The image generation unit 660 (see FIG. 15) may generate the highquality image 820-4 formed by further increasing the image quality of anarea closer to the center of the area 810-3. This can reduce thestrength of the high image quality formation for an area with a lowprobability of the existence of the moving object. Therefore, the amountof computation for the high image quality formation may be reducedcompared to when the image quality of the entire area is uniformlyincreased at a high strength.

FIG. 18 illustrates an example of a determination process of the featurearea by the second feature area specifying unit 620 described in FIG.15. To determine whether a specific image area 900 is a feature area,the second feature area specifying unit 620 extracts the amount offeatures from partial areas 910-1 to 4 in a predetermined positionalrelationship in the image area 900. At this time, the second featurearea specifying unit 620 extracts, from each of the partial areas 910,an amount of feature of a predetermined type according to the positionof each of the partial areas 910 in the image area 900.

The second feature area specifying unit 620 calculates, for each partialarea 910, a matching degree of the amount of features extracted from theimage of the partial area 910 relative to a predetermined condition. Thesecond feature area specifying unit 620 determines whether the imagearea 900 is a feature area based on the matching degree calculated foreach partial area 910. The second feature area specifying unit 620 maydetermine that the image area 900 is a feature area if the totalweighted value of the matching degrees is greater than a predeterminedvalue. The second feature area specifying unit 620 may also determinethat the image area 900 is a feature area if the number of partial areas910, in which the matching degree greater than a predetermined value iscalculated, is greater than a predetermined value.

The process from the extraction of the amount of features to thematching degree calculation may be implemented by an image filter. Theprocess may be implemented as a weak classifier. The positions of thepartial areas 910 may be determined according to the type of the objectto be extracted as the feature area. For example, when an area includingan object of the face of a person is to be detected as the feature area,the partial areas 910 may be determined at positions in whichdiscrimination power to the object of the face of the person is greaterthan a predetermined value. The high discrimination power may denotethat the probability that the determination result indicate “true” foran object of the face of the person is high and that the probabilitythat the determination result indicates “false” for objects other thanthe face of the person is high.

The image generation unit 660 (see FIG. 15) does not increase the imagequality of the areas other than the partial areas 910 and increases theimage quality of only the partial areas 910. As described, the secondfeature area specifying unit 620 extracts feature areas from the imagewith increased quality to determine whether the image area 900 is afeature area. This can increase the detection probability of the featureareas while limiting the image areas for which the image quality isincreased and can quickly detect the feature areas at a highprobability. Although the determination process of the feature areas inthe second feature area specifying unit 620 is described above, thefirst feature area specifying unit 610 may also determine whether theareas are the feature areas by the same process.

The processes in the first feature area specifying unit 610 and thesecond feature area specifying unit 620 can be implemented by aplurality of weak classifiers. An example of using and implementing Nweak classifiers in total will be described below. The first featurearea specifying unit 610 uses Nf weak classifiers to determine whetherthe areas are the feature areas. The matching degree is calculated basedon the determination result, and as described above, an area with thematching degree greater than the first threshold is determined as thefeature area, and an area with the matching degree equal to or smallerthan the second threshold is determined as a non-feature area.

The image generation unit 660 increases the image quality of the areaswith the matching degree equal to or smaller than the first thresholdand greater than the second threshold. The second feature areaspecifying unit 620 uses the Nf weak classifiers used by the firstfeature area specifying unit 610 and Nb weak classifiers other than theNf weak classifiers to determine whether the image with increasedquality includes feature areas. For example, whether the areas are thefeature areas may be determined based on the matching degrees calculatedfrom the determination results of the Nf+Nb weak classifiers.

Among the areas not specified as the feature areas by the first featurearea specifying unit 610, the feature areas may be specified bydifferent processes from a plurality of areas determined according tothe comparison results of a third threshold, which is smaller than thefirst threshold and greater than the second threshold, and the matchingdegrees. For example, the image generation unit 660 may increase theimage quality of the areas, in which the matching degree greater thanthe third threshold is calculated, and the Nf+Nb weak classifiers in thesecond feature area specifying unit 620 may determine whether the areasare the feature areas. Meanwhile, the image generation unit 660 mayincrease the image quality of the areas, in which the matching degreeequal to or smaller than the third threshold is calculated, and theNf+Nb weak classifiers in the second feature area specifying unit 620may determine whether the areas are the feature areas.

The number of weak classifiers Nb used in the process by the secondfeature area specifying unit 620 may be adjusted according to thematching degree. For example, if the matching degree is smaller, moreweak classifiers may be used in the second feature area specifying unit620 to determine whether the areas are the feature areas.

As described, the second feature area specifying unit 620 may search thefeature areas from the image quality change image in greater detail ifthe matching degree is lower. An example of a weak classifierconfiguration in at least one of the first feature area specifying unit610 and the second feature area specifying unit 620 includes a weakclassifier configuration by AdaBoost.

The first feature area specifying unit 610 and the second feature areaspecifying unit 620 may detect the feature areas from a low resolutionimage group formed by multiresolution expressions. In this case, theimage generation unit 660 may generate the low resolution image groupbased on highly accurate multiresolution formation by multiresolutionformation in the first feature area specifying unit 610. An example ofthe multiresolution formation process in the first feature areaspecifying unit 610 includes a reduction process by a bicubic method.

An example of the multiresolution formation process in the secondfeature area specifying unit 620 includes a reduction process based onprior studying. The second feature area specifying unit 620 may useimage processing parameters obtained by studying using full-scale imagesand images of target resolution to generate the low resolution imagegroup from the input images. It is more preferable to use images oftarget resolution with lower folding noise in the studying. For example,images obtained by different imaging devices including different numbersof imaging elements can be used for the studying.

The image processing method using the tensor projection of the presentinvention can be applied as the high image quality formation processdescribed in relation to FIGS. 15 to 18. More specifically, the imagegeneration unit 660 may use the image processing technique of high imagequality formation of the present invention illustrated in FIGS. 1 to 12to generate the high quality image formed by further increasing theimage quality of the areas with a high possibility of being specified asthe feature areas.

Other than the process of high resolution formation, examples of thehigh image quality formation process include a multi-gradation formationprocess for increasing the number of gradations and a multi-colorformation process for increasing the number of colors. The imageprocessing method using the tensor projection of the present inventioncan be applied to the processes.

If the photographic images subject to the high image quality formationare video configuration images (frame images or field images) of avideo, pixel values of other photographic images may be used to increasethe image quality in high image quality formation processes, such ashigh resolution formation, multi-color formation, multi-gradationformation, noise reduction, artifact reduction for reducing artifactssuch as block noise and mosquito noise, blur reduction, sharpnessformation, and high frame rate formation. For example, the image qualitymay be increased using the deviation of the imaging position of a movingbody due to a difference in the imaging timing. More specifically, theimage generation unit 660 may use photographic images as videoconfiguration images included in the video and other video configurationimages included in the video to generate high quality images.

Other than the process using a plurality of video configuration images,examples of the noise reduction process include processes described inJapanese Patent Application Laid-Open No. 2008-167949, Japanese PatentApplication Laid-Open No. 2008-167950, Japanese Patent ApplicationLaid-Open No. 2008-167948, and Japanese Patent Application Laid-Open No.2008-229161. For example, the image generation unit 660 can use a resultof prior studying using images with a larger amount of noise and imageswith a smaller amount of noise to reduce the noise. To reduce the amountof noise of the images taken by visible light as in the presentembodiment, images taken under the ambient light with a smaller amountlight can be used in the prior studying, instead of using images takenby small doses in the prior studying as described in Japanese PatentApplication Laid-Open No. 2008-167949. In the sharpness formationprocess, examples of more accurate sharpness formation process include aprocess using a filter in a larger filter size and a process of formingthe sharpness in more directions.

Example of Configuration of Compression Unit 232

FIG. 19 illustrates an example of a block configuration of thecompression unit 232 described in FIG. 14. The compression unit 232includes an image dividing unit 242, a plurality of fixed valueformation units 244 a-c (hereinafter, may be collectively called “fixedvalue formation units 244”), and a plurality of compression processingunits 246 a-d (hereinafter, may be collectively called “compressionprocessing units 246”).

The image dividing unit 242 acquires a plurality of photographic imagesfrom the image acquisition unit 222. The image dividing unit 242 dividesthe plurality of photographic images into feature areas and backgroundareas other than the feature areas. Specifically, the image dividingunit 242 divides the plurality of photographic images into a pluralityof individual feature areas and background areas other than the featureareas. The compression processing units 246 compress feature area imagesas images of the feature areas and background area images as images ofthe background areas at different strengths. Specifically, thecompression processing units 246 compress feature area videos includinga plurality of feature area images and background area videos includinga plurality of background area images at different strengths.

More specifically, the image dividing unit 242 generates feature areavideos of each type of a plurality of features by dividing the pluralityof photographic images. The fixed value formation units 244 fix thepixel values of the areas other than the feature areas of the types ofthe features in each feature area image included in the plurality offeature area videos generated for each type of features.

Specifically, the fixed value formation units 244 set predeterminedpixel values to the pixel values of the areas other than the featureareas. For each type of the features, the compression processing units246 a-c compress the plurality of feature area videos by MPEG or otherencoding formats.

The fixed value formation units 244 a-c fix the values of the featurearea video of a first feature type, the feature area video of a secondfeature type, and the feature area video of a third feature type,respectively. The compression processing units 246 a-c compress thefeature area video of the first feature type, the feature area video ofthe second feature type, and the feature area video of the third featuretype including the values fixed by the fixed value formation units 244a-c, respectively.

The compression processing units 246 a-c compress the feature areavideos at strengths predetermined according to the types of thefeatures. For example, the compression processing units 246 may convertthe feature area videos to videos with different resolutionspredetermined according to the types of the features of the featureareas to compress the converted feature area videos. Furthermore, in thecompression of the feature area videos by MPEG encoding, the compressionprocessing units 246 may compress the feature area videos by differentquantization parameters predetermined according to the types of thefeatures.

The compression processing unit 246 d compresses the background areavideos. The compression processing unit 246 d may compress thebackground area videos at a strength higher than the compressivestrengths of the compression processing units 246 a-c. The feature areavideos and the background area videos compressed by the compressionprocessing units 246 are supplied to the association processing unit 234(see FIG. 14).

As described in FIG. 19, the fixed value formation units 244 fix thevalues of the areas other than the feature areas. Therefore, when thecompression processing unit 246 performs predictive coding based on theMPEG encoding or the like, the amount of difference between the imagesand the prediction images in the areas other than the feature areas canbe significantly reduced. Therefore, the compression unit 232 cancompress the feature area videos at higher compression rates.

Although each of the plurality of compression processing units 246included in the compression unit 232 compresses the images of theplurality of feature areas and the images of the background areas in theconfiguration of FIG. 19, however, in another mode, the compression unit232 may include one compression processing unit 246, and the onecompression processing unit 246 may compress the images of the pluralityof feature areas and the images of the background areas at differentstrengths. For example, the images of the plurality of feature areas andthe images of the background areas may be sequentially supplied to theone compression processing unit 246 in a time division manner, and theone compression processing unit 246 may sequentially compress the imagesof the plurality of feature areas and the images of the background areasat different strengths.

The one compression processing unit 246 may also quantize the imageinformation of the plurality of feature areas and the image informationof the background areas by different quantization coefficients tocompress the images of the plurality of feature areas and the images ofthe background areas at different strengths. Images converted to imageswith different image qualities from the images of the plurality offeature areas and the images of the background areas may be supplied tothe one compression processing unit 246, and the one compressionprocessing unit 246 may compress the images of the plurality of featureareas and the images of the background areas. In the mode in which theone compression processing unit 246 quantizes the information bydifferent quantization coefficients area-by-area or compresses theimages converted to different image qualities area-by-area as describedabove, the one compression processing unit 246 may compress entire oneimage or may compress the images divided by the image dividing unit 242as described in FIG. 19. When the one compression processing unit 246compresses the entire one image, the dividing process by the imagedividing unit 242 and the fixed value formation process by the fixedvalue formation units 244 do not have to be executed. Therefore, thecompression unit 232 does not have to include the image dividing unit242 and the fixed value formation units 244.

Example of Configuration 2 of Compression Unit 232

FIG. 20 illustrates another example of a block configuration of thecompression unit 232 described in FIG. 14. The compression unit 232 inthe present configuration compresses a plurality of photographic imagesby a spatially scalable encoding process according to the type of thefeatures.

The compression unit 232 illustrated in FIG. 20 includes an imagequality conversion unit 510, a difference processing unit 520, and anencoding unit 530. The difference processing unit 520 includes aplurality of interlayer difference processing units 522 a-d(hereinafter, collectively called “interlayer difference processingunits 522”). The encoding unit 530 includes a plurality of encoders 532a-d (hereinafter, collectively called “encoders 532”).

The image quality conversion unit 510 acquires a plurality ofphotographic images from the image acquisition unit 222. The imagequality conversion unit 510 also acquires information for specifying thefeature areas detected by the feature area specifying unit 226 andinformation for specifying the types of the features of the featureareas. The image quality conversion unit 510 replicates the photographicimages to generate the same number of photographic images as the typesof the features of the feature areas. The image quality conversion unit510 converts the generated photographic images to images of resolutionscorresponding to the types of the features.

For example, the image quality conversion unit 510 generates aphotographic image converted to a resolution corresponding to thebackground area (hereinafter, called a “low resolution image”), aphotographic image converted to a first resolution corresponding to thetype of the first feature (hereinafter, called a “first resolutionimage”), a photographic image converted to a second resolutioncorresponding to the type of the second feature (hereinafter, called a“second resolution image”), and a photographic image converted to athird resolution corresponding to the type of the third feature(hereinafter, called a “third resolution image”). The first resolutionimage has a higher resolution than the low resolution image, the secondresolution image has a higher resolution than the first resolutionimage, and the third resolution image has a higher resolution than thesecond resolution image.

The image quality conversion unit 510 supplies the low resolution image,the first resolution image, the second resolution image, and the thirdresolution image to the interlayer difference processing unit 522 d, theinterlayer difference processing unit 522 a, the interlayer differenceprocessing unit 522 b, and the interlayer difference processing unit 522c, respectively. The image quality conversion unit 510 applies the imagequality conversion process to the plurality of photographic images andsupplies videos to the interlayer difference processing units 522.

The image quality conversion unit 510 may convert the frame rate of thevideos supplied to the interlayer difference processing units 522according to the types of the features of the feature areas. Forexample, the image quality conversion unit 510 may supply, to theinterlayer difference processing unit 522 d, a video with a frame ratelower than the video supplied to the interlayer difference processingunit 522 a. The image quality conversion unit 510 may also supply, tothe interlayer difference processing unit 522 a, a video at a frame ratelower than the video supplied to the interlayer difference processingunit 522 b and may supply, to the interlayer difference processing unit522 b, a video at a frame rate lower than the video supplied to theinterlayer difference processing unit 522 c. The image qualityconversion unit 510 may thin out the photographic images according tothe types of the features of the feature areas to convert the frame rateof the videos supplied to the interlayer difference processing units522.

The interlayer difference processing unit 522 d and the encoder 532 dapply predictive coding to the background area video including aplurality of low resolution images. Specifically, the interlayerdifference processing unit 522 generates difference images between theimages and prediction images generated from other low resolution images.The encoder 532 d quantizes a conversion factor obtained by convertingthe difference images to spatial frequency components and encodes thequantized conversion factor by entropy coding or the like. Thepredictive coding process may be applied to each partial area of the lowresolution images.

The interlayer difference processing unit 522 a applies predictivecoding to a first feature area video including a plurality of firstresolution images supplied from the image quality conversion unit 510.Similarly, the interlayer difference processing unit 522 b and theinterlayer difference processing unit 522 c apply predictive coding to asecond feature area video including a plurality of second resolutionimages and a third feature area video including a plurality of thirdresolution images, respectively. Hereinafter, specific operations of theinterlayer difference processing unit 522 a and the encoder 532 a willbe described.

The interlayer difference processing unit 522 a decodes the firstresolution images after encoding by the encoder 532 d and enlarges thedecoded images to images with the same resolution as the firstresolution. The interlayer difference processing unit 522 a generatesdifferent images between the enlarged images and the low resolutionimages. At this time, the interlayer difference processing unit 522 asets a difference value in the background area to 0. The encoder 532 aencodes the difference images in the same way as the encoder 532 d. Theencoding process by the interlayer difference processing unit 522 a andthe encoder 532 a may be applied to each partial area of the firstresolution images.

In the encoding of a first resolution image, the interlayer differenceprocessing unit 522 a compares an amount of encoding predicted when thedifference image between the first resolution image and the lowresolution image is encoded and an amount of encoding predicted when thedifference image between the first resolution image and the predictionimage generated from another first resolution image is encoded. If thelatter amount of encoding is smaller, the interlayer differenceprocessing unit 522 a generates a difference image between the firstresolution image and the prediction image generated from the other firstresolution image. If it is predicted that the amount of encoding issmaller when the image is encoded without calculating the differencebetween the first resolution image and the low resolution image or theprediction image, the interlayer difference processing unit 522 a maynot calculate the difference between the first resolution image and thelow resolution image or the prediction image.

The interlayer difference processing unit 522 a may not set thedifference value in the background area to 0. In this case, the encoder532 a may set the data after encoding of the difference information inthe areas other than the feature areas to 0. For example, the encoder532 a may set the conversion factor after conversion to the frequencycomponents to 0. Motion vector information when the interlayerdifference processing unit 522 d has performed the predictive coding issupplied to the interlayer difference processing unit 522 a. Theinterlayer difference processing unit 522 a may use the motion vectorinformation supplied from the interlayer difference processing unit 522d to calculate a motion vector for prediction image.

Operations of the interlayer difference processing unit 522 b and theencoder 532 b are substantially the same as the operations of theinterlayer difference processing unit 522 a and the encoder 532 a exceptthat the second resolution image is encoded and that the difference fromthe first resolution image after encoding by the encoder 532 a may becalculated when the second resolution image is encoded. Therefore, thedescription will not be repeated. Similarly, operations of theinterlayer difference processing unit 522 c and the encoder 532 c aresubstantially the same as the operations of the interlayer differenceprocessing unit 522 a and the encoder 532 a except that the thirdresolution image is encoded and that the difference from the secondresolution image after encoding by the encoder 532 b may be calculatedwhen the third resolution image is encoded. Therefore, the descriptionwill not be repeated.

As described, the image quality conversion unit 510 generates, from eachof the plurality of photographic images, a low quality image withreduced image quality and a feature area image with higher quality thanthe low quality image at least in the feature areas. The differenceprocessing unit 520 generates a feature area difference image indicatinga difference image between an image of a feature area in the featurearea image and an image of the feature area in the low quality image.The encoding unit 530 encodes the feature area difference image and thelow quality image.

The image quality conversion unit 510 generates low quality images withreduced resolution from the plurality of photographic images, and thedifference processing unit 520 generates feature area difference imagesbetween images of the feature areas in the feature area images andimages enlarged from the images of the feature areas in the low qualityimages. The difference processing unit 520 has spatial frequencycomponents obtained by converting the difference between the featurearea image in the feature area and the enlarged image to the spatialfrequency area and generates a feature area difference image in whichthe amount of data of the spatial frequency components is reduced in theareas other than the feature areas.

As described, the compression unit 232 encodes the differences of imagesbetween a plurality of layers with different resolutions to performhierarchical encoding. As is clear from this, it is clear that part ofthe system of compression by the compression unit 232 of the presentconfiguration includes a compression system based on H.264/SVC. Toexpand the layered compressed video, the image processing device 250decodes the video data of each layer, and for the areas encoded based onthe interlayer differences, executes an addition process with thephotographic images decoded in the layers in which the differences arecalculated. In this way, the photographic images with the originalresolution can be generated.

[Description of Image Processing Device 250]

FIG. 21 illustrates an example of a block configuration of the imageprocessing device 250 illustrated in FIG. 13. As illustrated in FIG. 21,the image processing device 250 includes a compressed image acquisitionunit 301, an association analysis unit 302, an expansion control unit310, an expansion unit 320, an external information acquisition unit380, and an image processing unit 330. The expansion unit 320 includes aplurality of decoders 322 a-d (hereinafter, collectively called“decoders 322”).

The compressed image acquisition unit 301 acquires the compressed videoscompressed by the image processing device 250. Specifically, thecompressed image acquisition unit 301 acquires the compressed videosincluding a plurality of feature area videos and background area videos.More specifically, the compressed image acquisition unit 301 acquiresthe compressed videos accompanied by the feature area information.

The association analysis unit 302 separates the compressed videos into aplurality of feature area videos as well as background area videos andthe feature area information and supplies the plurality of feature areavideos and background area videos to the expansion unit 320. Theassociation analysis unit 302 analyzes the feature area information andsupplies the positions of the feature areas and the types of thefeatures to the expansion control unit 310 and the image processing unit330.

The expansion control unit 310 controls an expansion process by theexpansion unit 320 according to the positions of the feature areas andthe types of the features acquired from the association analysis unit302. For example, the expansion control unit 310 causes the expansionunit 320 to expand the areas of the videos indicated by the compressedvideos according to the compression system used by the compression unit232 to compress the areas of the videos in accordance with the positionsof the feature areas and the types of the features.

The decoders 322 decode one of the plurality of encoded feature areavideos and background area videos. Specifically, the decoder 322 a, thedecoder 322 b, the decoder 322 c, and the decoder 322 d decode the firstfeature area video, the second feature area video, the third featurearea video, and the background area video, respectively.

The image processing unit 330 combines the plurality of feature areavideos and background area videos expanded by the expansion unit 320 andgenerates one video. Specifically, the image processing unit 330combines the images of the feature areas on the photographic imagesincluded in the plurality of feature area videos with the photographicimages included in the background area videos to generate one displayvideo. The image processing unit 330 may generate a display video inwhich the image quality of the feature areas is higher than the imagequality of the background areas. The super-resolution image processingmeans using the tensor projection of the present invention can be usedfor the conversion process of the high image quality formation.

The image processing unit 330 outputs the feature area information andthe display video acquired from the association analysis unit 302 to thedisplay device 260 or the image DB 255 (see FIG. 13). The positions ofthe feature areas indicated by the feature area information, the typesof the features of the feature areas, and the number of feature areasmay be associated with the information for identifying the photographicimages included in the display video, and the image DB 255 may berecorded in a non-volatile recording medium such as a hard disk.

The external information acquisition unit 380 acquires data used inimage processing by the image processing unit 330 from the outside ofthe image processing device 250. The image processing unit 330 uses thedata acquired by the external information acquisition unit 380 toexecute the image processing. The data acquired by the externalinformation acquisition unit 380 will be described in relation to FIG.22.

Example of Configuration of Image Processing Unit 330

FIG. 22 illustrates an example of a block configuration of the imageprocessing unit 330 included in the image processing device 250described in FIG. 21. As illustrated in FIG. 22, the image processingunit 330 includes a parameter storage unit 1010, an attribute specifyingunit 1020, a specific object area detection unit 1030, a parameterselection unit 1040, a weight determination unit 1050, a parametergeneration unit 1060, and an image generation unit 1070.

The parameter storage unit 1010 stores, in association with a pluralityof attributes of subject images, a plurality of image processingparameters for increasing the image quality of the subject images withthe attributes. The attribute specifying unit 1020 specifies theattributes of the subject images included in the input images. The inputimages may be the frame images obtained by the expansion unit 320. Theparameter selection unit 1040 prioritizes and selects the plurality ofimage processing parameters stored in the parameter storage unit 1010 inassociation with the attributes that are in more conformity with theattributes specified by the attribute specifying unit 1020. The imagegeneration unit 1070 uses the plurality of image processing parametersselected by the parameter selection unit 1040 to generate high qualityimages formed by increasing the image quality of the subject imagesincluded in the input images. The super-resolution image processingmeans using the tensor projection of the present invention is used forthe conversion process of the high image quality formation.

An example of the attributes includes a state of the subject, such asthe direction of the subject. More specifically, the parameter storageunit 1010 stores a plurality of image processing parameters inassociation with a plurality of attributes indicating the state of thesubject imaged as a subject image. The attribute specifying unit 1020specifies, from the subject image, the state of the subject imaged asthe subject image included in the input image.

An example of the state of the subject includes the direction of thesubject when the image is taken. The direction of the subject can be,for example, the direction of the face of a person as an example of thesubject. In this case, the parameter storage unit 1010 stores aplurality of image processing parameters in association with a pluralityof attributes indicating the direction of the subject taken as thesubject image. The attribute specifying unit 1020 specifies, from thesubject image, the direction of the subject taken as the subject imageincluded in the input image.

The attribute can also be the type of the subject. Examples of the typeof the subject include, sex of the person as the subject, age of theperson, expression of the imaged person, gesture of the imaged person,posture of the imaged person, race of the imaged person, wearingmaterial worn by the imaged person (such as glasses sunglasses, mask,and hat), and illumination state. The parameter storage unit 1010 maystore a plurality of image processing parameters in association with aplurality of attributes including at least one of the variousattributes. In this case, the attribute specifying unit 1020 specifies,from the subject image, the attributes of the person imaged as thesubject image included in the input image.

The weight determination unit 1050 determines weights for the pluralityof image processing parameters when the image quality of the subjectimage included in the input image is increased. Based on the weightsdetermined by the weight determination unit 1050, the image generationunit 1070 uses the plurality of image processing parameters selected bythe parameter selection unit 1040 to generate a high quality imageformed by increasing the image quality of the input image. The weightdetermination unit 1050 may determine a greater weight for an imageprocessing parameter associated with an attribute with a greatermatching degree with respect to the specified attribute.

The parameter generation unit 1060 generates combined parametersobtained by combining a plurality of image processing parametersselected by the parameter selection unit 1040. The image generation unit1070 uses the combined parameters generated by the parameter generationunit 1060 to increase the image quality of the subject image included inthe input image to generate a high quality image.

The generation of the image processing parameters according to theattributes of the subject is described above. The image processing unit330 may also change, on the image, the strength of the high imagequality formation.

The parameter storage unit 1010 stores specific parameters, which areimage processing parameters used to increase the image quality of theimages of a specific object, and non-specific parameters, which areimage processing parameters used to increase the image quality of theimages in which an object is not specified. As described later, thenon-specific parameters may be general-purpose image processingparameters with some advantageous effects of the high image qualityinformation regardless of the object.

The specific object area detection unit 1030 detects, from the inputimage, a specific object area that is an area of the specific object.The specific object may be an object of the subject that should bedetected as a feature area. The weight determination unit 1050determines weights of the specific parameters and the non-specificparameters in the high image quality formation of the input image fromwhich the specific object area is detected.

The weight determination unit 1050 determines a greater weight for thespecific parameters than for the non-specific parameters in the image ofthe specific object area in the input image. This can properly increasethe image quality of the specific object that should be detected as thefeature area. The weight determination unit 1050 determines a greaterweight for the non-specific parameters than for the specific parametersin the image of the non-specific object area that is an area other thanthe specified object area. This can prevent the high image qualityformation by the image processing parameters dedicated to the specificobject.

Based on the weights determined by the weight determination unit 1050,the image generation unit 1070 uses the specific parameters and thenon-specific parameters to generate a high quality image obtained byincreasing the image quality of the input image.

The parameter storage unit 1010 stores specific parameters calculated bystudying using a plurality of images of a specific object as studyingimages (also called “training images”) and non-specific parameterscalculated by studying using a plurality of images that are not imagesof the specific object as studying images. As a result, specificparameters specialized in the specific object can be calculated.General-purpose specific parameters for a variety of objects can also becalculated.

In the prior studying, it is desirable to study not the luminanceinformation of the studying images, but the image processing parametersusing spatial variation information such as edge information of thestudying images. The use of the edge information with reducedinformation of the low spatial frequency area can realize robust highimage quality formation processing for an illumination variation,particularly, with respect for an illumination change of a lowfrequency.

The parameter generation unit 1060 may generate combined parameters bycombining the non-specific parameters and the specific parameters by theweights determined by the weight determination unit 1050. The imagegeneration unit 1070 may use the combined parameters generated by theparameter generation unit 1060 to increase the image quality of theinput image to generate the high quality image.

In the example, an operation of generating the high quality image usinga plurality of image processing parameters selected based on theattributes of the subject specified by the attribute specifying unit1020 is described. The image generation unit 1070 may also use adifferent combination of the plurality of image processing parameters toincrease the image quality of the subject images included in the inputimages. For example, the image generation unit 1070 may use a differentcombination of a plurality of predetermined image processing parametersto increase the image quality of the subject images included in theinput images. The image generation unit 1070 may select at least one ofthe plurality of images obtained by increasing the image quality basedon the comparison with the input images to set the selected image as thehigh quality image. For example, the image generation unit 1070 mayprioritize and select, as the high quality image, an image with imagecontent more similar to the input images among the plurality of imagesobtained by increasing the image quality.

The parameter selection unit 1040 may select a different combination ofthe plurality of image processing parameters based on the attributes ofthe subject specified from the input images. The image generation unit1070 may use the selected plurality of image processing parameters toincrease the image quality of the subject images included in the inputimages. The image generation unit 1070 may select at least one of theplurality of images obtained by increasing the image quality based onthe comparison with the input images to set the selected image as thehigh quality image.

As described, even if the parameter storage unit 1010 stores a limitednumber of image processing parameters, the image processing device 250can increase the image quality using the image processing parametersthat can handle the images of the subject with a variety of attributes.Examples of the high image quality formation include high resolutionformation, multi-gradation formation, multi-color formation, as well asnoise reduction, artifact reduction, blur reduction, sharpnessformation, and high frame rate formation. The parameter storage unit1010 can store image processing parameters for the various high imagequality formation processes.

The external information acquisition unit 380 described in FIG. 21acquires the image processing parameters stored in the parameter storageunit 1010 (see FIG. 22) from the outside. The parameter storage unit1010 stores the image processing parameters acquired by the externalinformation acquisition unit 380. Specifically, the external informationacquisition unit 380 acquires at least one of the specific parametersand the non-specific parameters from the outside. The parameter storageunit 1010 stores at least one of the specific parameters and thenon-specific parameters acquired by the external information acquisitionunit 380.

FIG. 23 illustrates a table format of an example the parameters storedin the parameter storage unit 1010. The parameter storage unit 1010stores specific parameters A0, A1 . . . as image processing parametersfor faces of persons in association with the directions of the faces.The specific parameters A0 and A1 are calculated in advance by priorstudying in which the images of the corresponding directions of thefaces are handled as the studying images.

An example of a high resolution formation process based on weighting andaddition in pixel values of peripheral pixels of target pixel will beillustrated to describe a calculation process of the specific parametersA based on the prior studying. It is assumed here that a pixel value yof the target pixel is calculated by weighting and addition in pixelvalues x_(i) (wherein, i=1 to n) of n peripheral pixels. Morespecifically, it is assumed that y=Σ(w_(i)x_(i)). Here, Σ denotes anaddition throughout i. The character w_(i) denotes weighting factors forthe pixel values x_(i) of the peripheral pixels, and the weightingfactors w_(i) serve as the specific parameters A to be calculated in theprior studying.

It is assumed that in face images including imaged faces in a specificdirection are used as the studying images. If the pixel values of thetarget pixels of k-th (where k=1 to m) studying images are y_(k),y_(k)=Σw_(i)x_(ki) can be expressed. In this case, the weighting factorsw_(i) can be calculated by a computation process such as the leastsquares method. For example, w_(i) that substantially minimizes a squareof a vector, in which k-th elements e_(k) are indicated bye_(k)=y_(k)−Σ(w_(i)x_(ki)), can be calculated by a computation processsuch as the least squares method. The specific parameters Acorresponding to the face directions can be calculated by applying thecalculation process of the specific parameters to the face images of aplurality of face directions.

The parameter storage unit 1010 also stores non-specific parameters Bfor objects that are not faces of persons. The non-specific parameters Bare calculated in advance by prior studying in which images of a varietyof subjects are handled as the studying images. The non-specificparameters B can be calculated by the same prior studying process as thespecific parameters A. For example, the non-specific parameters B can becalculated using images other than persons, instead of face images, asthe studying images in the prior studying process for calculating thespecific parameters A.

FIG. 24 illustrates an example of weighting of a specific parameter. Itis assumed that an area 1210 and an area 1220 inside a heavy line in animage 1200 are detected as feature areas. In the area 1210 more insidebetween the feature areas, the weight determination unit 1050 (see FIG.22) determines that the weighting factor of the specific parameter is100% and the weighting factor of the non-specific parameter is 0%. Inthe area 1220 closer to the non-feature area outside of the area 1210 inthe feature areas (inside the heavy line frame), the weightdetermination unit 1050 determines that the weighting factor of thespecific parameter is 80% and the weighting factor of the non-specificparameter is 20%.

In an area 1230 near the feature areas in the area outside of thefeature areas, the weight determination unit 1050 determines that theweighting factor of the specific parameter is 50% and the weightingfactor of the non-specific parameter is 50%. In an area 1250 furtheroutside, the weight determination unit 1050 determines that theweighting factor of the specific parameter is 0% and the weightingfactor of the non-specific parameter is 100%.

In this way, the weight determination unit 1050 (see FIG. 22) determinesthe weights so that the weights for the specific parameters are greaterin the images of the areas more inside in the specific object areas ofthe input image. The weight determination unit 1050 also determines theweights so that the weights for the specific parameters are greater nearthe specific object areas in the images of the non-specific object areasthat are areas other than the specific object areas. In this way, theweight determination unit 1050 reduces the weighting factors of thespecific parameters step-by-step from the center of the feature areas tothe outside and from the feature areas to the non-feature areas. Otherthan the step-by-step reduction of the weighting factors, the weightdetermination unit 1050 may continuously reduce the weighting factors inproportion to the distance from the center of the feature areas or thedistance from the surrounding areas or the like of the feature areas.For example, the weight determination unit 1050 may determine theweighting factors with a value that decreases in terms of power orexponentially relative to a distance x, such as by reducing the value ofthe weighting factors relative to the distance x according to functionssuch as 1/x, 1/x², and e^(−x).

The weight determination unit 1050 may control the weighting factorsaccording to the detection reliability of the feature areas.Specifically, the weight determination unit 1050 sets greater weightsfor the specific parameters in the images of the specific object areaswith greater detection reliability of the areas of the specific objects.

If there is a specific object in an area not detected as a feature area,whether the specific object exists may not be able to be determined evenif the image quality of the area is increased by a general-purposenon-specific parameter. The image processing unit 330 executes a highimage quality formation process with an effect of a specific parameterfor specific object even in the area not detected as the feature area.Therefore, whether the specific object exists may be easily determinedfrom the high quality image.

The specific parameters may be image processing parameters obtained bycombining the plurality of image processing parameters described inrelation to FIG. 23. For example, it is assumed that a detected featurearea includes an image of the face of a person facing 15° to the sidefrom the front face. In this case, the weight determination unit 1050determines that the weighting factor for the specific parameter A0 is25% and determines that the weighting factor for the specific parameterA1 is 75%. The parameter generation unit 1060 generates a combinedparameter obtained by combining the specific parameter A0 and thespecific parameter A1 with the weighting factors 25% and 75%,respectively. The image generation unit 1070 uses an image processingparameter obtained by weighting the combined parameter generated by theparameter combining unit and the non-specific parameter at the rateillustrated in FIG. 24 to increase the image quality.

For example, when the image processing parameters (specific parametersor non-specific parameter) for increasing the image quality by weightingand addition in the peripheral pixels is used, the parameter generationunit 1060 may weight and add the weighting factor determined by theweight determination unit 1050 to the weighting factor of the imageprocessing parameter to calculate the combined parameter indicated bythe obtained weighting factor. Other than the weighting factor, examplesof the image processing parameter that can be added include spatialfrequency components in the spatial frequency area and image data (forexample, image data of the high frequency components).

Additionally, when the high image quality formation process is expressedby a vector computation, a matrix computation, or a tensor computationfor a feature amount vector and the like, the parameter generation unit1060 may generate the combined parameter by weighting and adding ormultiplying a vector, a matrix, a tensor, an n-dimensional mixed normaldistribution, or an n-dimensional mixed multinomial distribution as theimage processing parameters. Here, n is an integer one or greater. Forexample, interpolation of the vector in the feature vector space may beable to reduce the blur due to combining on the vector that cannot beexpressed by a scalar. For example, a computation of handling a sum of afeature vector obtained by multiplying a feature vector in a directionof 0° by a factor 0.25 and a feature vector obtained by multiplying afeature vector in a direction of 20° by a factor 0.75 as a featurevector in a direction of 15° can be illustrated as an example.Interpolation on the space of the locality preserving projection (LPP)may be able to further reduce the combined blur. The parametergeneration unit 1060 can calculate the combined parameter from thespecific parameter and the non-specific parameter. The parametergeneration unit 1060 can also calculate the combined parameter from aplurality of different specific parameters.

In the generation of a high quality image using the specific parameterand the non-specific parameter, the image generation unit 1070 maygenerate the high quality image by adding the image information obtainedby image processing using the specific parameter and the imageinformation obtained by image processing using the non-specificparameter based on the weighting factors determined by the weightdetermination unit 1050. Additionally, the image generation unit 1070may use the non-specific parameter to apply image processing to theimage information obtained by image processing using the specificparameter to generate the high quality image. A similar process can beapplied to a high image quality formation process using a plurality ofspecific parameters. Examples of the image data here include pixelvalues, a feature amount vector in the feature amount space, a matrix,an n-dimensional mixed normal distribution, and an n-dimensional mixedmultinomial distribution. For example, interpolation of the vector inthe feature vector space may be able to reduce the blur due to combiningon the vector that cannot be expressed by the scalar.

In the high image quality formation process described in FIGS. 23 and24, the parameter selection unit 1040 has selected the plurality ofimage processing parameters used to increase the image quality of thefeature areas based on the direction of the face of the person specifiedfrom the image in the feature areas. The image generation unit 1070 usesthe plurality of image processing parameters selected by the parameterselection unit 1040 to generate one high quality image.

Additionally, the image generation unit 1070 may generate a plurality ofimages formed by increasing the image quality of the feature areas froma plurality of combinations of the image processing parameters stored inthe image generation unit 1070. The image generation unit 1070 maygenerate a high quality image formed by increasing the image quality ofthe feature area from an image most similar to the image in the featurearea among the obtained plurality of images.

For example, the image generation unit 1070 uses the combined parameterof the specific parameter A0 corresponding to the direction of 0° andthe specific parameter A1 corresponding to the direction of 20° togenerate an image formed by increasing the quality of the image of thefeature area. The image generation unit 1070 further uses the combinedparameter of other one or more combinations of the specific parametersto generate one or more images formed by increasing the quality of theimage of the feature area.

The image generation unit 1070 compares the generated plurality ofimages with the image in the feature area to calculate a degree ofcoincidence of the image content. The image generation unit 1070determines the image with the highest degree of coincidence among thegenerated plurality of images as the high quality image.

In the generation of the plurality of images formed by increasing thequality of the image in the feature area, the image generation unit 1070may increase the image quality of the image in the feature area by aplurality of combined parameters based on a plurality of predeterminedsets of specific parameters. In this case, the parameter selection unit1040 may select the plurality of predetermined sets of specificparameters without the execution of the process of specifying thedirection of the face by the attribute specifying unit 1020.

Alternatively, the parameter selection unit 1040 may select a pluralityof sets of the specific parameters based on the direction of the face ofthe person specified from the image in the feature area. For example,the parameter selection unit 1040 associates and stores the informationfor specifying the plurality of sets of the specific parameters and theinformation for specifying the direction of the face of the person andmay select a plurality of sets of the specific parameters stored inassociation with the direction of the face of the person specified fromthe image in the feature area. The image quality of the image in thefeature area may be increased by the plurality of combined parametersbased on the plurality of selected sets to generate a plurality ofimages formed by increasing the image quality of the image of thefeature area.

In the generation of the plurality of images formed by increasing thequality of the image of the feature area, the image generation unit 1070may increase the quality of the image in the feature area based on theplurality of specific parameters. The image generation unit 1070 maygenerate an image most similar to the image in the feature area amongthe obtained plurality of images as the high quality image formed byincreasing the image quality of the feature area. In this case, theparameter selection unit 1040 may select a plurality of predeterminedspecific parameters without the execution of the process for specifyingthe direction of the face by the attribute specifying unit 1020, or theparameter selection unit 1040 may select a plurality of specificparameters based on the direction of the face of the person specifiedfrom the image in the feature area.

As described in relation to FIG. 23, the image processing parameter(specific parameter) for increasing the image quality of the face imagein a specific face direction can be calculated from the studying imagein the specific face direction. The image processing parameters can becalculated in the same way for other plurality of face directions tocalculate the image processing parameters corresponding to the pluralityof face directions. The parameter storage unit 1010 stores in advancethe calculated image processing parameters in association with thecorresponding face directions. Although the image processing parametersfor increasing the image quality of the face image may be imageprocessing parameters for increasing the image quality of the entireface, the image processing parameters may be image processing parametersfor increasing the image quality of at least part of the objectsincluded in the face images, such as an image of eyes, an image ofmouth, an image of nose, an image of ears.

The face direction is an example of the direction of the subject, and inthe same way as for the direction of the face, a plurality of imageprocessing parameters corresponding to the directions of a plurality ofsubjects can be calculated for the directions of other subjects. If thesubject is a person, the direction of a human body can be illustrated asthe direction of the subject, and more specifically, the direction oftorso, the direction of hands, and the like can be illustrated as thedirection of the human body. If the subject is not a person, a pluralityof image processing parameters for increasing the image quality of thesubject images with the imaged subject facing a plurality of directionscan be calculated in the same way as for the face images.

The direction of the subject is an example of the state of the subject,and the state of the subject can also be classified by expressions of aperson. In this case, the plurality of image processing parametersstored in the parameter storage unit 1010 increase the image quality ofimages of the face with different specific expressions. For example, theplurality of image processing parameters stored in the parameter storageunit 1010 increase the image quality of the faces when the person is inemotional states, the face when the person is in a nervous state, andthe like.

The state of the subject can also be classified by gestures of a person.In this case, the plurality of image processing parameters stored in theparameter storage unit 1010 increase the image quality of the images ofthe person in the state of different specific gestures. For example, theplurality of image processing parameters stored in the parameter storageunit 1010 increase the image quality of an image of a person running, animage of a person walking at a quick pace, an image of a person about torun, an image of a person hunting for a thing, and the like.

The state of the subject can also be classified by postures of a person.In this case, the plurality of image processing parameters stored in theparameter storage unit 1010 increase the image quality of the images ofthe person in the state of different specific postures. For example, theplurality of image processing parameters stored in the parameter storageunit 1010 increase the image quality of an image of a person bending theback, an image of a person putting hands in pockets, an image of aperson crossing the arms, an image of a person in which the directionsof the face and the body do not match, and the like.

The state of the subject can also be classified by wearing materials ofa person. In this case, the plurality of image processing parametersstored in the parameter storage unit 1010 increase the image quality ofthe images of the person in the state of wearing different specificwearing materials. For example, the image processing parameters storedin the parameter storage unit 1010 increase the image quality of animage of a person wearing glasses, an image of a person wearingsunglasses, an image of a person wearing a mask, an image of a personwearing a hat, and the like.

As described, the subject is classified into a plurality of attributesaccording to a plurality of states of the subject. The subject can alsobe classified into a plurality of attributes by the type of the subject.An example of the type of the subject includes the race of a person.Examples of the race of a person include a regionally classified race,such as an Asian race and a European race, and a physicalanthropologically classified race. In this case, the plurality of imageprocessing parameters stored in the parameter storage unit 1010 increasethe image quality of the images of persons classified into correspondingraces.

As for the type of the subject, classification by sex of the person,such as male and female, is possible. In this case, the plurality ofimage processing parameters stored in the parameter storage unit 1010increase the image quality of the images of the persons of thecorresponding sex, such as images of male or female. As for the type ofthe subject, classification by age groups of persons is possible. Inthis case, the plurality of image processing parameters stored in theparameter storage unit 1010 increase the image quality of the images ofthe persons of corresponding age, such as images of teenagers and imagesof people in their twenties.

The attributes of the subject images are defined by the illustratedtypes of subject, the plurality of states of subject, or a combinationof the types and the states. The parameter storage unit 1010 stores inadvance the image processing parameters for increasing the image qualityof the subject images belonged to the attributes in association with thedefined attributes. The image processing parameters stored in theparameter storage unit 1010 can be calculated by a similar method as thecalculation method of the image processing parameters for facedirection. For example, when the attributes are defined by expressions,a plurality of images with laughing faces can be studied in advance asstudying images to calculate the image processing parameters forincreasing the image quality of the images of laughing faces. Images ofother expressions, such as images of angry faces, can be similarlystudied in advance to calculate a plurality of image processingparameters for increasing the image quality of the face images of theexpressions. The image processing parameters can be similarly calculatedfor the attributes defined by gestures, postures, wearing materials,race, sex, age, and the like.

The attribute specifying unit 1020 can apply a classifier calculated inadvance by boosting, such as by AdaBoost, to the subject image tospecify the attributes of the subject image. For example, a plurality offace images with the face in a specific direction are used as theteacher images, and weak classifiers are integrated by a boostingprocess to generate the classifier. Whether the image is a face image ofthe specific face direction can be determined according to a right orwrong classification result obtained when the subject image is appliedto the generated classifier. For example, when a right classificationresult is obtained, the input subject image can be determined as theface image in the specific face direction.

A plurality of classifiers corresponding to the face directions can besimilarly generated for other plurality of face directions by generatingthe classifier by the boosting process. The attribute specifying unit1020 can apply the plurality of classifiers to the subject images tospecify the face directions based on right or wrong classificationresults obtained from the classifiers. Other than the face direction,the classifiers generated for the attributes by the boosting process canbe applied to specify one or more other attributes defined byexpressions, sex, and the like. Other than the studying by boosting, theattribute specifying unit 1020 can apply the classifiers studied for theattributes by various methods, such as a linear discrimination methodand a mixed Gaussian model, to the subject images to specify theattributes.

Example of Configuration of Display Device 260

FIG. 25 illustrates an example of a block configuration of the displaydevice 260 in FIG. 13. As illustrated in FIG. 25, the display device 260includes an image acquisition unit 1300, a first image processing unit1310, a feature area specifying unit 1320, a parameter determinationunit 1330, a display control unit 1340, a second image processing unit1350, an external information acquisition unit 1380, and a display unit1390.

The image acquisition unit 1300 acquires input images. The input imageshere may be frame images included in a video received from the imageprocessing device 250. The first image processing unit 1310 usespredetermined image processing parameters to generate predeterminedquality images obtained by increasing the image quality of the inputimages. For example, to increase the resolution, the first imageprocessing unit 1310 uses image processing parameters of a system inwhich the required amount of computation is smaller than a predeterminedvalue, such as a simple interpolation enlargement process, to generatethe predetermined quality images.

The display control unit 1340 displays the predetermined quality imagesgenerated by the first image processing unit 1310 on the display unit1390. In this way, the display unit 1390 displays the predeterminedquality images.

The feature area specifying unit 1320 specifies a plurality of featureareas in the input images. The feature area specifying unit 1320 mayspecify the plurality of feature areas in the input images when thedisplay unit 1390 displays the predetermined quality images. The imageprocessing device 250 may attach information for specifying the featureareas to a video as auxiliary information and transmit the informationto the display device 260. The feature area specifying unit 1320 mayextract the information for specifying the feature areas from theauxiliary information of the video acquired by the image acquisitionunit 1300 to specify the plurality of feature areas.

The parameter determination unit 1330 determines, for each of theplurality of feature areas, image processing parameters for furtherincreasing the image quality of the images of the plurality of featureareas. For example, the parameter determination unit 1330 determines,for each of the plurality of feature areas, the image processingparameters for increasing the image quality of the images of theplurality of feature areas at different strengths. “Increasing the imagequality at different strengths” may denote increasing the image qualitywith different amounts of computation, increasing the image quality withdifferent amounts of computation per unit area, or increasing the imagequality by high image quality formation systems with different requiredamounts of computation.

The second image processing unit 1350 uses the image processingparameters determined by the parameter determination unit 1330 togenerate a plurality of high quality feature area images formed byincreasing the image quality of the images of the plurality of featureareas. The display control unit 1340 displays the plurality of featurearea images in the plurality of feature areas in the predeterminedquality images displayed by the display unit 1390. In this way, thedisplay control unit 1340 displays the high quality images in place ofthe predetermined quality images already displayed by the display unit1390 when the high quality images are generated. Since the display unit1390 quickly generates and displays the predetermined quality images,the user can observe a monitoring video in certain image qualitysubstantially without a delay.

The parameter determination unit 1330 may determine the image processingparameters for the plurality of feature areas based on the importance ofthe images of the plurality of feature areas. Information indicating theimportance may be attached to the auxiliary information. The importancemay be predetermined according to the type of the subject of the featureareas. The user who observes the display unit 1390 may set theimportance of each type of the subject. The parameter determination unit1330 determines the image processing parameters for increasing the imagequality of the feature areas with greater importance at a higherstrength. Therefore, the user can observe images in which importantfeature areas have higher image quality.

The parameter determination unit 1330 determines the image processingparameters for each of the plurality of feature areas based on the typesof features of the images of the plurality of feature areas. Theparameter determination unit 1330 may also determine the imageprocessing parameters for each of the plurality of feature areas basedon the types of the subjects imaged in the plurality of feature areas.In this way, the parameter determination unit 1330 may directlydetermine the image processing parameters according to the types of thesubjects.

The parameter determination unit 1330 determines the image processingparameters based on the amount of process required to increase the imagequality of the plurality of feature areas in the second image processingunit 1350. Specifically, the parameter determination unit 1330determines the image processing parameters for increasing the imagequality at a higher strength when the required amount of process issmall.

For example, the parameter determination unit 1330 may determine theimage processing parameters for increasing the resolution at a higherstrength when the areas of the plurality of feature areas are small. Thesecond image processing unit 1350 uses the image processing parametersdetermined by the parameter determination unit 1330 to generate aplurality of high quality feature area images formed by increasing theresolutions of the images of the plurality of feature areas. Theparameter determination unit 1330 may determine the image processingparameters for increasing the image quality at a higher strength whenthe number of pixels of the plurality of feature areas is smaller.

The parameter determination unit 1330 determines the image processingparameters based on the processing capacity as an acceptable throughputin the second image processing unit 1350. Specifically, the parameterdetermination unit 1330 may determine the image processing parametersfor increasing the image quality at a higher strength when theprocessing capacity is smaller.

Therefore, the extent of the high image quality formation can becontrolled according to the amount of computation that can be processedby the second image processing unit 1350. Therefore, an excessive loadon the display unit 1390 by the high image quality formation process anda delay in the display of the images may be prevented. If there is aroom in the amount of computation by the display unit 1390, high qualityimages are quickly generated, and the images can be observed.

As described, the high resolution formation can be illustrated as thehigh image quality formation. Specifically, the parameter determinationunit 1330 determines, for each of the plurality of feature areas, theimage processing parameters for increasing the resolution of the imagesof the plurality of feature areas. The second image processing unit 1350uses the image processing parameters determined by the parameterdetermination unit 1330 to generate a plurality of high quality featurearea images formed by increasing the resolution of the images of theplurality of feature areas. The increase in the resolution at a highstrength includes highly accurately increasing the resolution andgenerating high quality images with more pixels.

Examples of the high image quality formation process include, inaddition to the high resolution formation, multi-gradation formation,multi-color formation process, noise reduction, artifact reduction, blurreduction, and sharpness formation. In the same way as for the highresolution formation, the parameter determination unit 1330 determines,for the various high image quality formations, the image processingparameters for various high image quality formations in each of theplurality of feature areas. The second image processing unit 1350 canuse the image processing parameters determined by the parameterdetermination unit 1330 to generate the plurality of high qualityfeature area images obtained by applying various high image qualityformations to the images of the plurality of feature areas.

As described, the image acquisition unit 1300 may acquire a plurality ofvideo configuration images included in a video as the input images. Theparameter determination unit 1330 determines, for each of the pluralityof feature areas, the image processing parameters for increasing theframe rate of the plurality of feature areas. The second imageprocessing unit 1350 may use the image processing parameters determinedby the parameter determination unit 1330 to generate a plurality of highquality feature area images formed by increasing the frame rate.

The parameter determination unit 1330 determines the image processingparameters based on the frame rate of the video. Specifically, theparameter determination unit 1330 may determine the image processingparameters for increasing the image quality at a higher strength whenthe frame rate of the video is smaller. The second image processing unit1350 may use the determined image processing parameters to increase theimage quality of the input images to generate a video formed byincreasing the image quality. As in the high image quality formation bythe image processing device 250, the high image quality formation by thesecond image processing unit 1350 may include concepts of highresolution formation, multi-color formation, multi-gradation formation,noise reduction, artifact reduction for reducing artifacts such as blocknoise and mosquito noise, blur reduction, and sharpness formation. Thesecond image processing unit 1350 can generate the high quality imagesby the processes.

In this way, the display device 260 can determine the strength of thehigh image quality formation according to the amount of data of theimages for high image quality formation and the amount of computationthat can be allocated to the high image quality formation process. Thedisplay device 260 can quickly provide images with a certain quality tothe user and can prevent a significant delay of the display of theimages subjected to the high image quality formation process. Therefore,the display device 260 can prevent the excessive load due to the highimage quality information process and can smoothly replay the videoprovided from the image processing device 250.

The external information acquisition unit 1380 acquires determinationconditions for determining the image processing parameters for thefeature areas from the outside of the display device 260. The parameterdetermination unit 1330 determines the image processing parameters forthe plurality of feature areas based on the determination conditionsacquired by the external information acquisition unit 1380. Examples ofthe determination conditions include parameters of the importance of thefeature areas, the type of the features of the feature areas, therequired amount of process, the area of the feature areas, the number ofpixels of the feature areas, and the processing capacity.

FIG. 26 illustrates an example of a display area 1400 of images. It isassumed that the display area 1400 is an area where the display unit1390 displays an input image. It is assumed here that three featureareas are specified from the input image. Images of the feature areasare displayed in a feature region area 1410, a feature region area 1420,and a feature region area 1430 of the display area 1400.

When the image acquisition unit 1300 described in FIG. 25 acquires aninput image, the display control unit 1340 displays the acquired inputimage on the display area 1400 of the display unit 1390.

While the input image is displayed, the second image processing unit1350 applies a predetermined high resolution formation process with arequired amount of computation smaller than a predetermined value, suchas simple interpolation, to the images of the feature areas to generatepredetermined quality images of the images of the feature areas (firsthigh resolution formation stage). As for the strength of the highresolution formation in the first high resolution formation stage, thesecond image processing unit 1350 applies a high resolution formationprocess at a predetermined strength regardless of the number of pixelsof the feature areas, the amount of data of images such as the framerate, the importance of the feature areas, the type of the subject, andthe computation tolerance in the second image processing unit 1350. Theamount of computation required to apply the high resolution formationprocess at the predetermined strength to the entire input image may bealways allocated to the second image processing unit 1350.

When a predetermined quality image 1412, a predetermined quality image1422, and a predetermined quality image 1432 are generated after thecompletion of the first high resolution formation stage, the displaycontrol unit 1340 displays the predetermined quality image 1412, thepredetermined quality image 1422, and the predetermined quality image1432 to the corresponding feature region area 1410, the feature regionarea 1420, and the feature region area 1430, respectively.

In a state that the predetermined quality image 1412, the predeterminedquality image 1422, and the predetermined quality image 1432 aredisplayed, the second image processing unit 1350 executes a highresolution formation process at strengths determined for the featureareas by the parameter determination unit 1330 to generate high qualityimages of the images of the feature areas (second high resolutionformation stage). In the second high resolution formation stage, thestrengths of the high resolution formation are strengths determined bythe parameter determination unit 1330, and the strengths of the highresolution formation depend on the amount of data of images such as thenumber of pixels of the feature areas and the frame rate, the importanceof the feature areas, the type of the subject, and the computationtolerance in the second image processing unit 1350.

When a high quality image 1414, a high quality image 1424, and a highquality image 1434 are generated after the completion of the second highresolution formation stage, the display control unit 1340 displays thehigh quality image 1414, the high quality image 1424, and the highquality image 1434 on the corresponding feature region area 1410, thefeature region area 1420, and the feature region area 1430,respectively.

In this way, the second image processing unit 1350 increases theresolution at a strength according to the amount of computation requiredfor the current amount of load and high image quality formation.Therefore, high quality images can be quickly provided to the userwithin a possible range.

Another Example of Embodiment of Image Processing System

FIG. 27 illustrates an example of the image processing system 201according to another embodiment. The configuration of the imageprocessing system 201 according to the present embodiment is the same asthe configuration of the image processing system 200 described in FIG.13, except that the imaging devices 210 a-d include image processingunits 804 a-d (hereinafter, collectively called “image processing units804”), respectively.

The image processing units 804 include constituent elements except theimage acquisition unit 222 among the constituent elements included inthe image processing device 220 described in FIG. 13. Functions andoperations of the constituent elements included in the image processingunits 804 may be substantially the same as the functions and theoperations of the constituent elements included in the image processingdevice 220 except that a video imaged by the imaging unit 212 isprocessed instead of the constituent elements included in the imageprocessing device 220 processing a video obtained by the compressedvideo expansion unit 224 through the expansion process. The sameadvantageous effects as the advantageous effects described in relationto the image processing system 200 from FIGS. 13 to 26 can also beobtained in the image processing system 201 with the configuration.

The image processing units 804 may acquire a video including a pluralityof photographic images expressed in an RAW format from the imaging unit212 to compress, in the RAW format, the plurality of photographic imagesexpressed in the RAW format included in the acquired video. The imageprocessing units 804 may detect one or more feature areas from theplurality of photographic images expressed in the RAW format. The imageprocessing units 804 may compress the video including the compressedplurality of photographic images in the RAW format. The image processingunits 804 can compress the video by the compression method described asan operation of the image processing device 220 in relation to FIGS. 13to 18. The image processing device 250 can expand the video acquiredfrom the image processing units 804 to acquire the plurality ofphotographic images expressed in the RAW format. The image processingdevice 250 enlarges, area-by-area, the plurality of photographic imagesexpressed in the RAW format acquired by the expansion and executes thesynchronization process area-by-area. At this time, the image processingdevice 250 may execute more accurate synchronization process in thefeature areas compared to the areas other than the feature areas.

The image processing device 250 may apply a super-resolution process tothe images of the feature areas in the photographic images obtained bythe synchronization process. The means of super-resolution using thetensor projection of the present invention can be applied as thesuper-resolution process in the image processing device 250.

The image processing device 250 may apply the super-resolution processto each object included in the feature areas. For example, when thefeature areas include face images of a person, the image processingdevice 250 applies the super-resolution process to each face region (forexample, eyes, nose, and mouth) as an example of the object. In thiscase, the image processing device 250 stores studying data, such as amodel described in Japanese Patent Application Laid-Open No.2006-350498, for each face region (for example, eyes, nose, and mouth).The image processing device 250 may use the studying data selected foreach face region included in the feature areas to apply thesuper-resolution process to the images of the face regions.

The studying data, such as a model, may be stored for each combinationof a plurality of expressions, a plurality of face directions, and aplurality of illumination conditions. Examples of the expressionsinclude faces in emotional states as well as a serious look, andexamples of the face direction include front, upward, downward,rightward, leftward, and backward. Examples of the illuminationconditions include conditions related to the illumination strength andthe direction of the illumination. The image processing device 250 mayuse the studying data corresponding to a combination of the faceexpressions, face directions, and illumination conditions to apply thesuper-resolution process to the face images.

The face expressions and the face directions can be specified based onthe image content of the face images included in the feature areas. Theexpression can be specified from the shape of the mouth and/or eyes, andthe face direction can be specified from the positional relationshipbetween the eyes, mouth, nose, and ears. The illumination strength tothe face and the illumination direction can be specified based on theimage content of the face image, such as the position and the size of ashade. The image processing units 804 specify the face expressions, theface directions, and the illumination conditions, and the output unit236 may transmit the specified face expressions, face directions, andillumination conditions in association with the images. The imageprocessing device 250 may use the studying data corresponding to theface expressions, the face directions, and the illumination conditionsreceived from the output unit 236 to execute the super-resolutionprocess.

Other than the model expressing the entire face, a model of each regionof the face can be used as the studying data, such as a model.Additionally, a model of the face of each sex and/or race can be used.The model can be stored not only for the persons, but also for each typeof objects to be monitored, such as vehicles and ships.

In this way, the image processing device 250 can use the localitypreserving projection (LPP) to reconfigure the images of the featureareas. Other than the locality preserving projection (LPP), othermethods for preserving the locality, such as locally linear embedding(LLE), can be used as an image reconfiguration method by the imageprocessing device 250 and a studying method for the imagereconfiguration.

In addition to the model described in Japanese Patent ApplicationLaid-Open No. 2006-350498, the studying data may include low frequencycomponents and high frequency components of the images of the objectsextracted from a multiplicity of sample images of the objects. The lowfrequency components of the images of the objects may be clustered bythe k-means method or the like for each type of the plurality of objectsto cluster the low frequency components of the images of the objectsinto a plurality of clusters for each type of the plurality of objects.A representative low frequency component (for example, a center ofgravity value) may be defined cluster by cluster.

The image processing device 250 extracts the low frequency componentsfrom the images of the objects included in the feature areas in thegraphic images. From the clusters of the low frequency componentsextracted from the sample images of the object of the type of theextracted object, the image processing device 250 specifies a cluster inwhich the value matching the extracted low frequency components isdefined as a representative low frequency component. The imageprocessing device 250 specifies a cluster of the high frequencycomponent associated with the low frequency component included in thespecified cluster. In this way, the image processing device 250 canspecify the clusters of the high frequency components correlated to thelow frequency components extracted from the object included in thephotographic images. The image processing device 250 may use a highfrequency component that represents the clusters of the specified highfrequency components to convert the image of the object to a highquality image with higher image quality. For example, the imageprocessing device 250 may add, to the image of the object, the highfrequency component selected for each object with a weight according tothe distance from the center of each object to the position to beprocessed on the face. The representative high frequency component maybe generated by closed-loop studying. In this way, the image processingdevice 250 selects and uses, for each object, desirable studying datafrom the studying data generated by studying in each object, and theimage processing device 250 may be able to increase the image quality ofthe images of the object at a higher accuracy.

The image processing device 250 can also use the stored low frequencycomponents and high frequency components to increase the image qualityof the input images without clustering by the k-means method and thelike. For example, the image processing device 250 stores pairs of lowresolution edge components as the edge components extracted from thepatches in low resolution studying images and high resolution edgecomponents as the edge components extracted from the patches in highresolution studying images. The edge components may be stored as vectorson the eigenspace of the LPP and the like.

To increase the image quality of the input image as a target of the highimage quality formation, the image processing device 250 extracts theedge components in each patch from an enlarged image obtained byenlarging the input image by a predetermined method such as bicubic. Theimage processing device 250 calculates, on the eigenspace of the LPP andthe like, norms between the extracted edge components and the storededge components for each patch in the input image. The image processingdevice 250 selects, from the stored patches, a plurality of patches inwhich norms smaller than a predetermined value are calculated. For thetarget patches and the surrounding patches, the image processing device250 sets a Markov random field of the extracted edge components and thehigh resolution edge components of the plurality of selected patches.The image processing device 250 uses loopy belief propagation (LBP) andthe like to solve an energy minimization problem of the Markov randomfield model set for each target patch to select, from the stored highresolution edge components and for each target patch, the highresolution edge components to be added to the image in each targetpatch. The image processing device 250 adds the high resolution edgecomponents selected in the patches to the image components of thepatches of the enlarged image to generate a high quality image.

Additionally, the image processing device 250 can also use a Gaussianmixture model of a plurality of classes to increase the image quality ofthe input images. For example, image vectors of the patches in the lowresolution studying images and the image vectors of the patches in thehigh resolution studying images are set as studying data. Clustervectors obtained from the image vectors of the patches in the lowresolution studying images are used to calculate an average and adispersion of the density distribution corresponding to the classes inthe Gaussian mixture model and weights for the classes based on an EMalgorithm and the like. The image processing device 250 stores theaverage, the dispersion, and the weights as studying data. To increasethe image quality of the input images as targets of the high imagequality formation, the image processing device 250 uses the imagevectors of the patches in the input images, the cluster vectors obtainedfrom the image vectors, as well as the average, the dispersion, and theweights stored as the studying data to generate the high quality images.

Additionally, the image processing device 250 can also use contourinformation extracted from an input image to generate high qualityimages only from the input image. For example, to increase theresolution of specific image areas near the contour extracted from theinput image, the image processing device 250 can arrange, in thespecific image areas, pixel values of the pixels included in other areasalong the contour to generate the high quality image formed byincreasing the resolution of the specific image areas. For example, theimage processing device 250 can determine on which positions in thespecific image areas the pixel values of the pixels will be arrangedbased on the positional relationship between the positions of the pixelsincluded in the other areas and the position of the contour and arrangethe pixel values on the determined positions to increase the resolutionof the specific image areas.

The image processing device 250 may execute the high resolutionformation process using the contour information only near the edge areasincluding the edge in the input image. The resolution may be increasedby a filtering system and the like in image areas other than the edgeareas. For example, the image processing device 250 may use thefiltering system to increase the resolution in flat areas from which anamount of edge smaller than a predetermined amount is extracted. Intexture areas from which an amount of edge greater than a predeterminedamount is extracted, the image processing device 250 may increase theresolution by correcting the image formed by increasing the resolutionusing the filter system so that the conditions generated from the inputimage are satisfied.

As described, the high image quality formation process using the lowfrequency components and the high frequency components, the Gaussianmixture model, and the high resolution formation process using thecontour information can be used to increase the image quality of theimage in which an object is not specified. The parameter storage unit1010 can store the parameters used in the high image quality formationby the image processing device 250, such as the data of the highfrequency components corresponding to the low frequency components, thefilters for increasing the resolution of the flat areas, the studyingdata related to the Gaussian mixture model, and the like. The high imagequality formation process using the locality preserving projectiontensor of the present invention can be applied as the process forincreasing the image quality of the image in which the object isspecified.

An example of the high image quality formation process for face imageswill be described as the high image quality formation process using thetensor. Face images with different resolutions, persons, and patchpositions are used as the studying images for calculating, by studying,a fourth-order tensor including resolutions, patch positions, persons,and pixels as studying targets. The studying images are used tocalculate eigenvectors in the eigenspace by targeting the resolutions,the patch positions, the persons, and the pixel values. The fourth-ordertensor based on the products of the calculated eigenvector is used togenerate medium resolution face images from the face images included inthe input images. The eigenvectors can be calculated by studying basedon the eigenvalue decomposition method, the locality preservingprojection (LPP), and the like. High resolution patches used to recoverthe high frequency components from the medium-resolution face images canbe obtained from the high resolution studying images. The imageprocessing device 250 stores the obtained tensors and high resolutionpatches.

When increasing the image quality of the face images included in theinput images as the targets of the high image quality formation, theimage processing device 250 converts the face images patch-by-patchbased on the stored fourth-order tensor to obtain the patches that formthe medium resolution face images. The image processing device 250 setsa Markov random field of the medium resolution patches and the storedhigh resolution patches. The energy minimization problem of all patchesin the Markov random field model is solved by an iterative correctionmethod (ICM) and the like to obtain high resolution face images in whichthe high frequency components are recovered.

When the configuration of the image processing device 100 described inFIG. 6 is adapted as the means for increasing the image quality in theimage processing device 250, the output images of the addition unit 160(or the combining unit 166) of FIG. 6 correspond to the “mediumresolution” face images. The “medium resolution” images are furtherinput to solve the energy minimization problem of the Markov randomfield model to obtain an output of the “high resolution” images.

The image processing device 250 may execute a process of generating lowresolution face images from the face images included in the input imagesas preprocessing for obtaining the medium resolution patches. In thiscase, the image processing device 250 converts, by the fourth-ordertensor, the low resolution face images obtained by the preprocessing toobtain the medium resolution patches. The preprocessing can include aprocess of converting the face images included in the input images byusing a fifth-order tensor obtained by targeting the directions offaces, illumination levels, expressions, persons, and pixels. Faceimages with different directions of faces, illumination levels,expressions, and persons can be used as the studying images forobtaining the fifth-order tensor.

It is desirable to include a positioning process of the face imagesincluded in the input images as the preprocessing. For example, the faceimages may be positioned by an affine transformation. More specifically,parameters of the affine transformation are optimized to match thepositions of the face images after the affine transformation and theface images for studying. It is obviously desirable to execute apositioning process for the face images for studying so that thepositions of the images match.

An example of the high image quality formation process using thelocality preserving projection (LPP) will be described below. In thestudying stage, eigenvectors are calculated by the locality preservingprojection (LPP) from the low resolution images and the high resolutionimages as the studying images. In the LPP space, the low resolutionimages and the high resolution images are associated by radial basisfunctions as weights of the network. Residual images between the mediumresolution images obtained from an input of the low resolution images ofthe studying images and the low resolution images as well as residualimages between the high resolution images of the studying images and themedium resolution images are calculated. The image processing device 250stores the residual images between the medium resolution images and thelow resolution images and the residual images between the highresolution images and the medium resolution images patch-by-patch.

When increasing the image quality of the input images as the targets ofthe high image quality formation, the image processing device 250generates the eigenvectors from the input image based on the localitypreserving projection (LPP) and the medium resolution images from theradial basis functions obtained in the studying stage. The imageprocessing device 250 calculates the residual images between the mediumresolution images and the input face images. The residual images betweencorresponding high resolution images and medium resolution images areselected patch-by-patch from the stored residual images based on thelocally linear embedding (LLE) and the nearest search. The imageprocessing device 250 adds residual images obtained by smoothing theresidual images between the selected high resolution images and mediumresolution images to the medium resolution images generated from theinput images to generate the high quality images.

In the super-resolution process based on the principal componentanalysis as described in Japanese Patent Application Laid-Open No.2006-350498, the images of the objects are expressed by principalcomponent vectors and weighting factors. The amount of data of theweighting factors and the principal component vectors is significantlysmall compared to the amount of data of the pixel data included in theimages of the objects. Therefore, the image processing units 804 maycalculate the weighting factors from the images of the objects includedin the feature areas in the compression process for compressing theimages of the feature areas in the plurality of photographic imagesacquired from the imaging unit 212. More specifically, the imageprocessing units 804 can compress the images of the objects included inthe feature areas by expressing the images by the principal componentvectors and the weighting factors. The image processing units 804 maytransmit the principal component vectors and the weighting factors tothe image processing device 250. In this case, the image processingdevice 250 can use the principal component vectors and the weightingfactors acquired from the image processing units 804 to reconfigure theimages of the objects included in the feature areas. Other than themodel based on the principal component analysis as described in JapanesePatent Application Laid-Open No. 2006-350498, it is obvious that theimage processing units 804 can use a model for expressing the objects byvarious feature parameters to compress the images of the objectsincluded in the feature areas.

In the configuration of the image processing system 10 described inrelation to FIGS. 1 to 14, the image processing device 250 or thedisplay device 260 can also apply the super-resolution process to theimages of the feature areas as the high image quality formation process.In the image processing system 10 and the image processing system 20,the compression unit 232 can, like the image processing device 220,express the images by the principal component vectors and the weightingfactors to further compress the photographic images.

An example of the monitoring system is illustrated above to describe theoperations of the image processing systems 200 and 201. A high imagequality formation process and encoding for a document scanned by ascanner device such as a copy machine can be applied as anotherapplication of the invention. For example, assuming that areas, such astexts, drawings, tables, and photographs, are feature areas, a highimage quality formation process, such as the super-resolution process,can be applied as the high resolution formation process to the areas.The feature area detection process and the compression process can beapplied to the detection and encoding of the feature areas. Similarly,the feature area detection process, the high image quality formationprocess, and the compression process can be applied to the detection,high image quality formation, and encoding of regions inside the body inan endoscopic system.

Modified Example 1

Although the image processing systems 200 and 201 include the pluralityof imaging devices 210 a-d in the example described above, the number ofimaging devices 210 is not particularly limited, and the number ofimaging devices 210 may be one. The number of display devices 260 is notparticularly limited either, and the number of display devices 260 maybe one.

Modified Example 2

Although the feature areas are specified from the photographic images(frame images or field images) in the video data in the image processingsystems 200 and 201, the image processing systems 200 and 201 can beapplied not only to the video data, but also to still image data.

Modified Example 3

Although the configuration that can detect a plurality of feature areasfrom one photographic image is described in the image processing systems200 and 201, the number of feature areas is not particularly limited,and there may be one feature area in one photographic image.

Modified Example 4

The mode of the means for acquiring the studying image group is notlimited to the mode of preparing the image group of pairs of highquality images and low quality images. Only the high quality images maybe provided, and the low quality images may be generated from the highquality images to obtain the image pairs. For example, processing means(low image quality formation processing means) for executing a processof decreasing the image quality may be included in the image processingdevice. High quality studying images can be input, and the device candecrease the image quality of the high quality studying images toacquire studying image pairs.

In the case of the image processing systems 200 and 201 described inFIGS. 13 and 27, the mode is not limited to the mode in which thestudying images are provided from a prepared database and the like. Thesystems can be operated to update the studying content based on imagesactually imported by the imaging device 210 or images (partial images)cut out from the images. The conversion accuracy can be further improvedby appropriately importing the studying images according to theapplication of the system or the installation location of the imagingdevice to execute the studying step again.

Modified Example 5

Although an example of studying image data to convert images byincreasing the image quality is described in the embodiment, the presentinvention can be applied not only to the high image quality formationprocess, but also to other image conversions such as image recognition.The data to be processed is not limited to the images, but various dataother than the images can be similarly applied. More specifically, theconfigurations described as the image processing device, the imageprocessing means, and the image processing system can be expanded as adata processing device, data processing means, and a data processingsystem.

<Application to Image Recognition>

An application to a technique of personal authentication based on imagerecognition will be described as an application other than the highimage quality formation process. In this case, the same process as theprocess up to the intermediate eigenspace of the high image qualityformation process described in FIGS. 2, 3, 6, and the like can beexecuted, and the positional relationship between the coefficientvectors in the intermediate eigenspace can be used to perform thepersonal authentication. The distances, the directions, and the like canbe obtained for the positional relationship based on the obtainingmethod of the “coefficient vector correction processing unit 140”. Morespecifically, the closer the obtained distances and directions of theinput data to the studying data, the higher the possibility that theinput data is a determination target.

Therefore, the similarity to a specific person (for example, likelihoodof “Person A”) can be determined from the positional relationshipbetween the studying data and the newly input data in the intermediateeigenspace (personal different eigenspace here).

Various conditions can be considered for the input images of faces, suchas facing to the front, to the left, to the right . . . , and a newadvantageous effect that one or more conditions can be accuratelyhandled by a single standard can be obtained using a property that evenif images of any directions are input, the images gather into one pointon the intermediate eigenspace (for example, personal differenceeigenspace) through a direction modality such as facing to the front, tothe left, to the right . . . .

Not only the modality of “direction”, but also resolution modalities oflow resolution, medium resolution, high resolution, . . . and the likeas well as various modalities described above can be similarly handled.In this way, the same applies to other modalities with one or moreconditions. Even if images of any conditions are input in relation to aspecific modality, one or more conditions can be accurately handled by asingle standard using the property that the images gather into one pointon the intermediate eigenspace through the specific modality.

<Application to Speech Recognition>

An example of an application to speech recognition will be described asan example of handling data other than the images. Voice data istargeted in place of the image data, and the process similar to theprocess up to the intermediate eigenspace of the high image qualityformation process described in FIGS. 2, 3, 6, and the like can beexecuted to perform the speech recognition using the positionalrelationship between the coefficient vectors in the intermediateeigenspace. The distances, the directions, and the like can be obtainedfor the positional relationship by the obtaining method of the“coefficient vector correction processing unit 140”. More specifically,the closer the obtained distances and directions of the input data tothe studying data, the higher the possibility that the input data is adetermination target.

In this case, for example, modalities of the number of sound samples(low resolution and high resolution) of the voice data are applied tothe pixel modalities (low resolution and high resolution) described forthe image data. A signal-to-noise ratio (S/N) and the positions of asound source and a microphone (sensor) can also be handled as themodalities.

In the conventional method, the studying eigenspace for speechrecognition needs to be prepared for each sampling frequency, such as 48kHz, 44.1 kHz, and 32 kHz, or each number of quantization hits, such as16 bits and 8 bits.

Conversely, according to the present invention, the determination ismade on a common studying eigenspace for speech recognition (equivalentto the “intermediate eigenspace”), and a plurality of samples andquantization bits can be commonly recognized and handled by one type ofcriteria. Therefore, there is an advantageous effect that the criteriado not have to be adjusted case by case. Moreover, the influence ofdegradation caused by a disturbance or noise included in the lowfrequency components can be removed by controlling the low frequencycomponents of the input to perform the tensor projection, and therobustness (strength) of the process for the low frequency components(such as a disturbance and noise) can be increased. Furthermore, theconversion by the projection preserving the local structure, such as inthe LPP using the local relationship, facilitates preserving the mediumfrequency components and the high frequency components that can beeasily lost in global information in the PCA and the like. Therefore, anew advantageous effect that there is a possibility of further improvingthe performance is obtained. The same advantageous effect can beobtained for the modalities such as the S/N and the sound sourcemicrophone position.

<Application to Language Processing>

An example of an application to language processing will be described asanother example of handling data other than the images. Language data(may be voice data or text data) is targeted in place of the image data,and the process similar to the process up to the intermediate eigenspaceof the high image quality formation process described in FIGS. 2, 3, 6,and the like can be executed to perform the language processing usingthe positional relationship between the coefficient vectors in theintermediate eigenspace. The distances, the directions, and the like canbe obtained for the positional relationship by the obtaining method ofthe “coefficient vector correction processing unit 140”. Morespecifically, the closer the obtained distances and directions of theinput data to the studying data, the higher the possibility that theinput data is a determination target.

In this case, for example, language (Japanese and English) modalitiesare applied to the pixel modalities (low resolution and high resolution)described in relation to the image data. Regions (dialects),applications (formal (news) and informal), eras (Heian period, Edoperiod, and modern era), and generations (high school students andelderly) can also be handled as the modalities.

In the conventional method, the studying eigenspace for languagerecognition needs to be prepared for each language, such as Japanese andEnglish.

Conversely, according to the present invention, the determination ismade on a common studying eigenspace for language recognition(equivalent to the “intermediate eigenspace”), and a plurality oflanguages can be commonly recognized and handled by one type ofcriteria. Therefore, there is an advantageous effect that the criteriado not have to be adjusted case by case. Moreover, the influence ofdegradation caused by a disturbance or noise included in the lowfrequency components can be removed by controlling the low frequencycomponents of the input to perform the tensor projection, and therobustness (strength) of the process for the low frequency components(such as a disturbance and noise) can be increased. Furthermore, theconversion by the projection preserving the local structure, such as inthe LPP using the local relationship, facilitates preserving the mediumfrequency components and the high frequency components that can beeasily lost in global information in the PCA and the like. Therefore, anew advantageous effect that there is a possibility of further improvingthe performance is obtained. The same advantageous effect can beobtained for the modalities such as the regions, the applications, theeras, and the generations.

<Application to Living Body Information Processing>

An example of an application to living body information processing willbe described as another example of handling data other than the images.Examples of the living body information include waveforms, periods, andamplitudes of the heartbeat, pulse, blood pressure, breathing, andsweating. The data of the living body information is targeted in placeof the image data, and the process similar to the process up to theintermediate eigenspace of the high image quality formation processdescribed in FIGS. 2, 3, 6, and the like can be executed to execute theliving body information process using the positional relationshipbetween the coefficient vectors in the intermediate eigenspace. Thedistances, the directions, and the like may be obtained for thepositional relationship by the obtaining method of the “coefficientvector correction processing unit 140”. More specifically, the closerthe obtained distances and directions of the input data to the studyingdata, the higher the possibility that the input data is a determinationtarget.

In this case, for example, modalities of the number of data samples (lowdecomposition and high decomposition) of the living body information areapplied to the pixel modalities (low resolution and high resolution)described in relation to the image data. The signal-to-noise ratio (S/N)and the positions of the signal source and the sensor can also behandled as the modalities.

In the conventional method, the studying eigenspace for living bodyinformation needs to be prepared for each sampling frequency or eachnumber of quantization bits.

Conversely, according to the present invention, the determination ismade on a common studying eigenspace for living body information process(equivalent to the “intermediate eigenspace”), and a plurality ofsamples and quantization bits can be commonly recognized and handled byone type of criteria. Therefore, there is an advantageous effect thatthe criteria do not have to be adjusted case by case. Moreover, theinfluence of degradation caused by a disturbance or noise included inthe low frequency components can be removed by controlling the lowfrequency components of the input to perform the tensor projection, andthe robustness (strength) of the process for the low frequencycomponents (such as a disturbance and noise) can be increased.Furthermore, the conversion by the projection preserving the localstructure, such as in the LPP using the local relationship, facilitatespreserving the medium frequency components and the high frequencycomponents that can be easily lost in global information in the PCA andthe like. Therefore, a new advantageous effect that there is apossibility of further improving the performance is obtained. The sameadvantageous effect can be obtained for the modalities such as the S/Nand the sensor position.

<Application to Natural/Physical Information Processing>

An example of an application to natural/physical information processingwill be described as another example of handling data other than theimages. Examples of the natural/physical information include waveforms,periods, and amplitudes of weather, climate, and earthquake. Data of thenatural/physical information is targeted in place of the image data, andthe same process as the process up to the intermediate eigenspace of thehigh image quality formation process described in FIGS. 2, 3, 6, and thelike can be executed to process the natural/physical information usingthe positional relationship between the coefficient vectors in theintermediate eigenspace. The distances, the directions, and the like maybe obtained for the positional relationship by the obtaining method ofthe “coefficient vector correction processing unit 140”. Morespecifically, the closer the obtained distances and directions of theinput data to the studying data, the higher the possibility that theinput data is a determination target.

In this case, for example, modalities of the number of data samples (lowdecomposition and high composition) are applied to the pixel modalities(low resolution and high resolution) described in relation to the imagedata. The signal-to-noise ratio (S/N) and the positions of the signalsource and the sensor can also be handled as the modalities.

In the conventional method, the studying eigenspace for natural/physicalinformation processing needs to be prepared for each sampling frequencyand each number of quantization bits.

Conversely, according to the present invention, the determination ismade on a common studying eigenspace for natural/physical informationprocessing (equivalent to the “intermediate eigenspace”), and aplurality samples and quantization bits can be commonly recognized andhandled by one type of criteria. Therefore, there is an advantageouseffect that the criteria do not have to be adjusted case by case.Moreover, the influence of degradation caused by a disturbance or noiseincluded in the low frequency components can be removed by controllingthe low frequency components of the input to perform the tensorprojection, and the robustness (strength) of the process for the lowfrequency components (such as a disturbance and noise) can be increased.Furthermore, the conversion by the projection preserving the localstructure, such as in the LPP using the local relationship, facilitatespreserving the medium frequency components and the high frequencycomponents that can be easily lost in global information in the PCA andthe like. Therefore, a new advantageous effect that there is apossibility of further improving the performance is obtained. The sameadvantageous effect can be obtained for the modalities such as the S/Nand the sensor position.

REFERENCE SIGNS LIST

100 . . . image processing device, 102 . . . low resolution enlargementprocessing unit, 104 . . . high pass filter, 108 . . . LPP projectiontensor generation unit, 115 LPP eigenprojection matrix, 116 . . . LPPprojection core tensor, 122 . . . first sub-core tensor generation unit,124 . . . second sub-core tensor generation unit, 130 . . . firstLPP_HOSVD projection processing unit, 150 . . . second LPP_HOSVDprojection unit, 160 . . . addition unit, 200 . . . image processingsystem, 201 . . . image processing system, 610 . . . first feature areaspecifying unit, 620 . . . second feature area specifying unit, 230 . .. compression control unit, 232 . . . compression unit

The invention claimed is:
 1. An image processing device characterized bycomprising: an information acquisition unit configured to acquire aneigenprojection matrix generated by a projection computation from astudying image group including at least one of an image pair formed byhigh frequency components of a first-quality image and a second-qualityimage with different image qualities and an image pair formed by thehigh frequency components and medium frequency components of thefirst-quality image and the second-quality image, and to acquire aprojection core tensor generated from the studying image group and theeigenprojection matrix; a first sub-core tensor generation unitconfigured to generate a first sub-core tensor corresponding to acondition specified by a first setting from the acquired projection coretensor; a second sub-core tensor generation unit configured to generatea second sub-core tensor corresponding to a condition specified by asecond setting from the acquired projection core tensor; a filteringunit configured to generate a low frequency component control image inwhich high frequency components or the high frequency components andmedium frequency components of an input image to be processed areextracted; a first subtensor projection unit configured to project thelow frequency component control image by a first projection computationusing the eigenprojection matrix and the first sub-core tensor tocalculate a coefficient vector in an intermediate eigenspace; a secondsubtensor projection unit configured to project the calculatedcoefficient vector by a second projection computation using the secondsub-core tensor and the eigenprojection matrix to generate a projectionimage from the low frequency component control image; an imageconversion unit configured to generate a conversion image with an imagequality different from the input image; and an addition unit configuredto add the projection image and the conversion image.
 2. An imageprocessing device comprising: an information acquisition unit configuredto acquire an eigenprojection matrix generated by a projectioncomputation from a studying image group including at least one of animage pair formed by high frequency components of a first-quality imageand a second-quality image with different image qualities and an imagepair formed by the high frequency components and medium frequencycomponents of the first-quality image and the second-quality image, toacquire a first sub-core tensor corresponding to a condition specifiedby a first setting, the first sub-core tensor generated using aprojection core tensor generated from the studying image group and theprojection matrix, and to acquire a second sub-core tensor correspondingto a condition specified by a second setting, the second sub-core tensorgenerated using the projection core tensor; a filtering unit configuredto generate a low frequency component control image in which highfrequency components or the high frequency components and mediumfrequency components of an input image to be processed are extracted; afirst subtensor projection unit configured to project the low frequencycomponent control image by a first projection computation using theeigenprojection matrix and the first sub-core tensor to calculate acoefficient vector in an intermediate eigenspace; a second subtensorprojection unit configured to project the calculated coefficient vectorby a second projection computation using the second sub-core tensor andthe eigenprojection matrix to generate a projection image from the lowfrequency component control image; an image conversion unit configuredto generate a conversion image with an image quality different from theinput image; and an addition unit configured to add the projection imageand the conversion image.
 3. The image processing device according toclaim 1, wherein the information acquisition unit acquires theeigenprojection matrix generated by the projection computation from thestudying image group including the image pair formed by the highfrequency components of the first-quality image and the second-qualityimage and acquires the projection core tensor generated from thestudying image group and the eigenprojection matrix, the filtering unitgenerates a high frequency component image in which the high frequencycomponents of the input image are extracted, and the first subtensorprojection unit, which is for projecting the low frequency componentcontrol image by the first projection computation using theeigenprojection matrix and the first sub-core tensor to calculate thecoefficient vector in the intermediate eigenspace, and the secondsubtensor projection unit generate a projection image of the highfrequency components from the high frequency component image to generateimage information of a high frequency area exceeding a frequency areaexpressed in the input image.
 4. An image processing device comprising:an eigenprojection matrix generation unit configured to generate aneigenprojection matrix generated by a projection computation from astudying image group including at least one of an image pair formed byhigh frequency components of a first-quality image and a second qualityimage with different image qualities and an image pair formed by thehigh frequency components and medium frequency components of thefirst-quality image and the second-quality image; a projection coretensor generation unit configured to generate a projection core tensordefining a correspondence between the high frequency components of thefirst-quality image and an intermediate eigenspace or between the highfrequency components as well as the medium frequency components of thefirst-quality image and the intermediate eigenspace and a correspondencebetween the high frequency components of the second-quality image andthe intermediate eigenspace or between the high frequency components aswell as the medium frequency components of the second-quality image andthe intermediate eigenspace; a first sub-core tensor acquisition unitconfigured to generate a first sub-core tensor corresponding to acondition specified by a first setting from the generated projectioncore tensor; a second sub-core tensor acquisition unit configured togenerate a second sub-core tensor corresponding to a condition specifiedby a second setting from the generated projection core tensor; afiltering unit configured to generate a low frequency component controlimage in which high frequency components or the high frequencycomponents and medium frequency components of an input image to beprocessed are extracted; a first subtensor projection unit configured toproject the low frequency component control image by a first projectioncomputation using the eigenprojection matrix and the first sub-coretensor to calculate a coefficient vector in the intermediate eigenspace;a second subtensor projection unit configured to project the calculatedcoefficient vector by a second projection computation using the secondsub-core tensor and the eigenprojection matrix to generate a projectionimage from the low frequency component control image; an imageconversion unit configured to generate a conversion image with an imagequality different from the input image; and an addition unit configuredto add the projection image and the conversion image.
 5. The imageprocessing device according to claim 4, wherein the eigenprojectionmatrix generation unit generates the eigenprojection matrix by theprojection computation from the studying image group including the imagepair formed by the high frequency components of the first-quality imageand the second-quality image, the projection core tensor generation unitgenerates the projection core tensor from the studying image group andthe eigenprojection matrix, the filtering unit generates a highfrequency component image in which the high frequency components of theinput image are extracted, and the first subtensor projection unit,which is for projecting the low frequency component control image by thefirst projection computation using the eigenprojection matrix and thefirst sub-core tensor to calculate the coefficient vector in theintermediate eigenspace, and the second subtensor projection unitgenerate the projection image of the high frequency components from thehigh frequency component image to generate image information of a highfrequency area exceeding a frequency area expressed in the input image.6. The image processing device according to claim 1, wherein the highfrequency components and the medium frequency components of thefirst-quality image are extracted by applying the same process as thefiltering unit to the first-quality image, and the high frequencycomponents and the medium frequency components of the second-qualityimage are extracted by applying the same process as the filtering unitto the second-quality image.
 7. The image processing device according toclaim 1, further comprising a weighting factor determination unitconfigured to determine weighting factors for weighting the projectionimage and the conversion image to be added by the addition unit.
 8. Theimage processing device according to claim 1, wherein the filtering unitexecutes a process of extracting components greater than a frequencybased on a Nyquist frequency in the input image.
 9. The image processingdevice according to claim 1, wherein the first-quality image is an imagewith a relatively low image quality of the image pair, thesecond-quality image is an image with a relatively high image quality ofthe image pair, and the conversion image is an image with a higher imagequality than the input image.
 10. The image processing device accordingto claim 1, wherein the first setting designates a projectionrelationship of projecting the first-quality image on the intermediateeigenspace, and the second setting designates a projection relationshipof projecting the second-quality on the intermediate eigenspace.
 11. Theimage processing device according to claim 1, wherein the projectioncomputation is one of locality preserving projection (LPP), locallylinear embedding (LLE), and linear tangent-space alignment (LTSA). 12.The image processing device according to claim 1, wherein the studyingimage group includes the image pair targeting the face of a person, andthe intermediate eigenspace is a personal difference eigenspace.
 13. Theimage processing device according to claim 1, further comprising: afirst feature area specifying unit configured to specify first featureareas from an inputted image; a compression processing unit configuredto compress image parts of the first feature areas in the inputted imageat a first compressive strength and compressing image parts of otherthan the first feature areas at a second compressive strength which is acompressive strength higher than the first compressive strength; and animage quality change processing unit configured to protect at least thefirst feature areas by the first subtensor projection unit and thesecond subtensor projection unit to change the image quality.
 14. Theimage processing device according to claim 1, wherein the projectioncomputation includes a projection computation using a localrelationship.
 15. An image processing method comprising: an informationacquisition step of acquiring an eigenprojection matrix generated by aprojection computation from a studying image group including at leastone of an image pair formed by high frequency components of afirst-quality image and a second-quality image with different imagequalities and an image pair formed by the high frequency components andmedium frequency components of the first-quality image and thesecond-quality image and acquiring a projection core tensor generatedfrom the studying image group and the eigenprojection matrix; a firstsub-core tensor generation step of generating a first sub-core tensorcorresponding to a condition specified by a first setting from theacquired projection core tensor; a second sub-core tensor generationstep of generating a second sub-core tensor corresponding to a conditionspecified by a second setting from the acquired projection core tensor;a filtering process step of generating a low frequency component controlimage in which high frequency components or the high frequencycomponents and medium frequency components of an input image to beprocessed are extracted; a first subtensor projection step of projectingthe low frequency component control image by a first projectioncomputation using the eigenprojection matrix and the first sub-coretensor to calculate a coefficient vector in an intermediate eigenspace;a second subtensor projection step of projecting the calculatedcoefficient vector by a second projection computation using the secondsub-core tensor and the eigenprojection matrix to generate a projectionimage from the low frequency component control image; an imageconversion step of generating a conversion image with an image qualitydifferent from the input image; and an addition step of adding theprojection image and the conversion image.
 16. An image processingmethod comprising: an information acquisition step of acquiring aneigenprojection matrix generated by a projection computation from astudying image group including at least one of an image pair formed byhigh frequency components of a first-quality image and a second-qualityimage with different image qualities and an image pair formed by thehigh frequency components and medium frequency components of thefirst-quality image and the second-quality image, acquiring a firstsub-core tensor corresponding to a condition specified by a firstsetting, the first sub-core tensor generated using a projection coretensor generated from the studying image group and the projectionmatrix, and acquiring a second sub-core tensor corresponding to acondition specified by a second setting, the second sub-core tensorgenerated using the projection core tensor; a filtering process step ofgenerating a low frequency component control image in which highfrequency components or the high frequency components and mediumfrequency components of an input image to be processed are extracted; afirst subtensor projection step of projecting the low frequencycomponent control image by a first projection computation using theeigenprojection matrix and the first sub-core tensor to calculate acoefficient vector in an intermediate eigenspace; a second subtensorprojection step of projecting the calculated coefficient vector by asecond projection computation using the second sub-core tensor and theeigenprojection matrix to generate a projection image from the lowfrequency component control image; an image conversion step ofgenerating a conversion image with an image quality different from theinput image; and an addition step of adding the projection image and theconversion image.
 17. An image processing method characterized bycomprising: an eigenprojection matrix generation step of generating aneigenprojection matrix generated by a projection computation from astudying image group including at least one of an image pair formed byhigh frequency components of a first-quality image and a second-qualityimage with different image qualities and an image pair formed by thehigh frequency components and medium frequency components of thefirst-quality image and the second-quality image; a projection coretensor generation step of generating a projection core tensor defining acorrespondence between the high frequency components of thefirst-quality image and an intermediate eigenspace and a correspondencebetween the high frequency components of the second-quality image andthe intermediate eigenspace; a first sub-core tensor acquisition step ofgenerating a first sub-core tensor corresponding to a conditionspecified by a first setting from the generated projection core tensor;a second sub-core tensor acquisition step of generating a secondsub-core tensor corresponding to a condition specified by a secondsetting from the generated projection core tensor; a filtering processstep of generating a low frequency component control image in which highfrequency components or the high frequency components and mediumfrequency components of an input image to be processed are extracted; afirst subtensor projection step of projecting the low frequencycomponent control image by a first projection computation using theeigenprojection matrix and the first sub-core tensor to calculate acoefficient vector in the intermediate eigenspace; a second subtensorprojection step of projecting the calculated coefficient vector by asecond projection computation using the second sub-core tensor and theeigenprojection matrix to generate a projection image from the lowfrequency component control image; an image conversion step ofgenerating a conversion image with an image quality different from theinput image; and an addition step of adding the projection image and theconversion image.
 18. The image processing method according to claim 15,wherein the projection computation includes a projection computationusing a local relationship.
 19. A non-transitory computer-readablerecording medium including a program stored thereon, such that when theprogram is read and executed by a computer, the computer is caused tofunction as: an information acquisition unit configured to acquire aneigenprojection matrix generated by a projection computation from astudying image group including at least one of an image pair formed byhigh frequency components of a first-quality image and a second-qualityimage with different image qualities and an image pair formed by thehigh frequency components and medium frequency components of thefirst-quality image and the second-quality image and acquiring aprojection core tensor generated from the studying image group and theeigenprojection matrix; a first sub-core tensor generation unitconfigured to generate a first sub-core tensor corresponding to acondition specified by a first setting from the acquired projection coretensor; a second sub-core tensor generation unit configured to generatea second sub-core tensor corresponding to a condition specified by asecond setting from the acquired projection core tensor; a filteringunit configured to generate a low frequency component control image inwhich high frequency components or the high frequency components andmedium frequency components of an input image to be processed areextracted; a first subtensor projection unit configured to project thelow frequency component control image by a first projection computationusing the eigenprojection matrix and the first sub-core tensor tocalculate a coefficient vector in an intermediate eigenspace; a secondsubtensor projection unit configured to project the calculatedcoefficient vector by a second projection computation using the secondsub-core tensor and the eigenprojection matrix to generate a projectionimage from the low frequency component control image; an imageconversion unit configured to generate a conversion image with an imagequality different from the input image; and an addition unit configuredto add the projection image and the conversion image.
 20. Anon-transitory computer-readable recording medium including a programstored thereon, such that when the program is read and executed by acomputer, the computer is caused to function as: an informationacquisition unit configured to acquire an eigenprojection matrixgenerated by a projection computation from a studying image groupincluding at least one of an image pair formed by high frequencycomponents of a first-quality image and a second quality image withdifferent image qualities and an image pair foiined by the highfrequency components and medium frequency components of thefirst-quality image and the second-quality image, to acquire a firstsub-core tensor corresponding to a condition specified by a firstsetting, the first sub-core tensor generated using a projection coretensor generated from the studying image group and the projectionmatrix, and to acquire a second sub-core tensor corresponding to acondition specified by a second setting, the second sub-core tensorgenerated using the projection core tensor; a filtering unit configuredto generate a low frequency component control image in which highfrequency components or the high frequency components and mediumfrequency components of an input image to be processed are extracted; afirst subtensor projection unit configured to project the low frequencycomponent control image by a first projection computation using theeigenprojection matrix and the first sub-core tensor to calculate acoefficient vector in an intermediate eigenspace; a second subtensorprojection unit configured to project the calculated coefficient vectorby a second projection computation using the second sub-core tensor andthe eigenprojection matrix to generate a projection image from the lowfrequency component control image; an image conversion unit configuredto generate a conversion image with an image quality different from theinput image; and an addition unit configured to add the projection imageand the conversion image.
 21. A non-transitory computer-readablerecording medium including a program stored thereon, such that when theprogram is read and executed by a computer, the computer is caused tofunction as: an eigenprojection matrix generation unit configured togenerate an eigenprojection matrix generated by a projection computationfrom a studying image group including at least one of an image pairformed by high frequency components of a first-quality image and asecond quality image with different image qualities and an image pairformed by the high frequency components and medium frequency componentsof the first-quality image and the second-quality image; a projectioncore tensor generation unit configured to generate a projection coretensor defining a correspondence between the high frequency componentsand an intermediate eigenspace or between the high frequency componentsas well as the medium frequency components and the intermediateeigenspace of the first-quality image and a correspondence between thehigh frequency components and the intermediate eigenspace or between thehigh frequency components as well as the medium frequency components andthe intermediate eigenspace of the second-quality image; a firstsub-core tensor acquisition unit configured to generate a first sub-coretensor corresponding to a condition specified by a first setting fromthe generated projection core tensor; a second sub-core tensoracquisition unit configured to generate a second sub-core tensorcorresponding to a condition specified by a second setting from thegenerated projection core tensor; a filtering unit configured togenerate a low frequency component control image in which high frequencycomponents or the high frequency components and medium frequencycomponents of an input image to be processed are extracted; a firstsubtensor projection unit configured to project the low frequencycomponent control image by a first projection computation using theeigenprojection matrix and the first sub-core tensor to calculate acoefficient vector in the intermediate eigenspace; a second subtensorprojection unit configured to project the calculated coefficient vectorby a second projection computation using the second sub-core tensor andthe eigenprojection matrix to generate a projection image from the lowfrequency component control image; an image conversion unit configuredto generate a conversion image with an image quality different from theinput image; and an addition unit configured to add the projection imageand the conversion image.
 22. The recording medium according to claim19, wherein the projection computation includes a projection computationusing a local relationship.
 23. A data processing device comprising: aninformation acquisition unit configured to acquire an eigenprojectionmatrix generated by a projection computation from a studying data groupincluding at least a data pair formed by medium frequency components orhigh frequency components of first-condition data and second-conditiondata with different conditions and to acquire a first sub-core tensorcreated corresponding to a condition specified by a first setting, thefirst sub-core tensor created from a projection core tensor that isgenerated from the studying data group and the eigenprojection matrixand that defines a correspondence between the first-condition data andan intermediate eigenspace and a correspondence between thesecond-condition data and the intermediate eigenspace; a filtering unitconfigured to generate low frequency component control input data inwhich high frequency components or the high frequency components andmedium frequency components of input data to be processed are extracted;and a first subtensor projection unit configured to project the lowfrequency component control input data by a first projection computationusing the eigenprojection matrix and the first sub-core tensor acquiredfrom the information acquisition unit to calculate a coefficient vectorin the intermediate eigenspace.
 24. The data processing device accordingto claim 23, wherein the projection computation includes a projectioncomputation using a local relationship.
 25. A data processing methodcomprising: an information acquisition step of acquiring aneigenprojection matrix generated by a projection computation from astudying data group including at least a data pair formed by mediumfrequency components or high frequency components of first-conditiondata and second-condition data with different conditions and acquiring afirst sub-core tensor created corresponding to a condition specified bya first setting, the first sub-core tensor created from a projectioncore tensor that is generated from the studying data group and theeigenprojection matrix and that defines a correspondence between thefirst-condition data and an intermediate eigenspace and a correspondencebetween the second-condition data and the intermediate eigenspace; afiltering step of generating low frequency component control input datain which high frequency components or the high frequency components andmedium frequency components of input data to be processed are extracted;and a first subtensor projection step of projecting the low frequencycomponent control input data by a first projection computation using theeigenprojection matrix and the first sub-core tensor acquired in theinformation acquisition step to calculate a coefficient vector in theintermediate eigenspace.
 26. The data processing method according toclaim 25, wherein the projection computation includes a projectioncomputation using a local relationship.
 27. A non-transitorycomputer-readable recording medium including a program stored thereon,such that when the program is read and executed by a computer, thecomputer is caused to function as: an information acquisition unitconfigured to acquire an eigenprojection matrix generated by aprojection computation from a studying data group including at least adata pair formed by medium frequency components or high frequencycomponents of first-condition data and second-condition data withdifferent conditions and to acquire a first sub-core tensor createdcorresponding to a condition specified by a first setting, the firstsub-core tensor created from a projection core tensor that is generatedfrom the studying data group and the eigenprojection matrix and thatdefines a correspondence between the first-condition data and anintermediate eigenspace and a correspondence between thesecond-condition data and the intermediate eigenspace; a filtering unitconfigured to generate low frequency component control input data inwhich high frequency components or the high frequency components andmedium frequency components of input data to be processed are extracted;and a first subtensor projection unit configured to project the lowfrequency component control input data by a first projection computationusing the eigenprojection matrix and the first sub-core tensor acquiredfrom the information acquisition unit to calculate a coefficient vectorin the intermediate eigenspace.
 28. A non-transitory computer-readablerecording medium including a program stored thereon, such that when theprogram is read and executed by a computer, the computer is caused tofunction as: an information acquisition unit configured to acquire aneigenprojection matrix generated by a projection computation from astudying data group including at least a data pair formed by mediumfrequency components or high frequency components of first-conditiondata and second-condition data with different conditions and to acquirea first sub-core tensor created corresponding to a condition specifiedby a first setting, the first sub-core tensor created from a projectioncore tensor that is generated from the studying data group and theeigenprojection matrix and that defines a correspondence between thefirst-condition data and an intermediate eigenspace and a correspondencebetween the second-condition data and the intermediate eigenspace; afiltering unit configured to generate low frequency component controlinput data in which high frequency components or the high frequencycomponents and medium frequency components of input data to be processedare extracted; and a first subtensor projection unit configured toproject the low frequency component control input data by a firstprojection computation using the eigenprojection matrix and the firstsub-core tensor acquired from the information acquisition unit tocalculate a coefficient vector in the intermediate eigenspace.
 29. Therecording medium according to claim 27, wherein the projectioncomputation includes a projection computation using a localrelationship.